Dodital Trek and Uttaranchal: Day 8 (15 Jun 08)

This entry is part 9 of 13 in the series Himalayan Trip: Dodital Trek and Uttaranchal

Day 8, 15 Jun 08

To Gangotri

3.00 am, Its a curse, the alarm is a god-damn curse on the human race. Nobody is really interested to getup and be ready for the great Gangotri trip, one of the चार धाम places. I’m unable to download anything, how on earth will bowels work at 3.30am in the morning. By 4.30 we’re partially ready when the driver knocks at the door. Everybody is extremely slow to move ahead, but finally by 4.45am we are in the Maxx Jeep and heading towards Gangotri. Pratap Rana, our driver moves the vehicle inside the Uttarkashi market place and moves his pawn by saying, he is getting a ‘savari’ or another person in the cab. Mallik is fumed, he doesn’t not wont anyone else in here. There is a lot of heated discussion between us and Rana. He suggests that he can give back the advance and drop us by the market taxi stand. But somehow we continue. But he is rash in his driving, moving hard on turns, on potholes. That irks all of us. But we stay quiet, Amey does not he starts another argument with him on his driving. Rana backfires, that this is the only place where he can swiftly move his jeep, ’cause in the ghats it has to be slow and steady.

7.00 am, We are still climbing through NH 108, moving through hilly villages overlooking the valley in which Bhagirathi is raging like a mad river. I recollect my mother’s stories about some place where the river flow is so hard that it can break anything in its path. We try to gauge the speed of the river it could be more than 40kmph. That means its an extremely heavy slope towards Uttarkashi and the river does not spare anything in its way. The mountains themselves are big, overpowering, strong and hard. Even though they are beautiful there is something hideous about them, something like ‘Get off my way’ attitude. Another hour or later we are near Bhatwari. And there seems to be a massive jam. Rains and day’s sun are the miscreants. There has been a lot of landslide further in the distance, hence blocking the traffic on either ends. Just if the natural forces were not enough for the problem, there was rock drilling continuing in the rock for better roads. The vibrations just help the brittle rock to dislocate. We get down and walk on foot to see how far is the jam. It could more than a km. long. We still walk ahead and find the actual place where the landslide has occurred. The landslide has taken off 20-30% of the road into the valley. The road around too is very mucky, slushy. No firm grip around. There are still small pebbles falling down. A brave truck driver helps push big boulders on the road with his Truck, and we have some space to move. Rana calls us and we all run back. Rana, true to the character of a local driver in the Himalayan ghats, manoeuvres his jeep through the jam, quickly getting ahead of big buses and other lazy cars. He says its important to get ahead, cause another jam and we would be late. It is necessary for us to leave early from Gangotri as well and clear through these ghats in the sunlight. We had moved a little ahead of the problem and there was another minor jam, thanks to a Maruti 800 which was unable to move the puddle of soil, gravel. The driver was trying hard to move the vehicle, but his efforts were in vain, A young local driver jumped off from his cab and offered to drive the car through the muck, and asked some others for a push at specific places. And brilliantly he got the small-tired car from foot deep soil.

Harsil

Its close to 10.30 and we are now close to Harsil, a beautiful hilly village before Gangotri, no other human settlement between Harsil and Gangotri, except for ITBP or the Indo Tibet Border Police checkposts. Harsil is beautiful, amidst the cool weather, High himalayas and lovely Apple Orchids that it harvests. We halted at a local ‘tapri’ for some food and chai. Most have ‘chole’ and ‘bun-pav’. Rnana urges us to be quick, so after a sip of tea, we are moving again. The climb continues and we spot our first snow-capped mountain in this area. They seem to be towering above us, the rain clouds trying to cover the snow. It was an amazing experience being in such a place and all cameras went snapping.

Its 11.00, and we are on a bridge, joining tow mountains, some 600m a river is flowing. Pratap Rana stops and narrates that this is the highest bridge over here, we are near Nelong Kopang, the Indo Tibet Border which is just few minutes away from here. Below the bridge he continues is the rivulet that is coming from China/Tibet which further joins the Bhagirathi coming from Gangotri, and he points to a far corner where the two rivers met. We continue from the place and soon we spot cars parked on either side of the road. The familiar parking problem in pilgrimage spots is no stranger here. We move on close to the actual place where Rana parks and says we should be running back by 1.30. It has taken us well over 4.5 hrs from Uttarkashi to Gangotri which is about 96km by roads. 99% of which are in high in Ghats.

Gangotri

The street leading to the temple is a typical one, with shops selling religious clothes, puja equipment and more. Mallik, Vj buy some yellow-saffron colored ‘gamcha’. The pressure inside my stomach is now building. As we enter the main area, we can hear water falling from somewhere, excited we move ahead towards Surya Kund, Where a small off-rivulet of Bhagirathi is jumping from a vertical cliff from about 30m and making a small pool below, the Surya Kund. The pressure, water flow is so much that it creates a mist of water in the surroundings. Videos and photos never far away from such places. We move towards the right climbing ahead on the path which has many aashrams, and other religious installations. Now I’m not really interested in all those, all I need is a toilet ! We cross through a wooden bridge to the other side, after I reached the other side of the bridge, we see a simple sign near the edge of bridge, कमज़ोर ब्रिज ! the whole troop now moves towards the ghats where people are trying to take a dip into the holy or rather chilly water. Even though Mallik has some fever, he decides that he’ll take a dip. Its close to noon and Vj joins him in the chill-party and we then head towards the temple. After a brief visit to the temple we are back on the main street. My friends are now finding a place to put more stuff inside their stomach, while I’m trying to find a place to do the exact opposite. Its now uncontrollable, while they settle for a hotel, I go ahead in search for a toilet, luckily I find a decently clean ‘Pay and Use Sulab Shochalay’. Aaaaa…hhh !!! I managed a Vartak Memorial at Gangotri. Light as a bird I join others for lunch.

Its 1.30-1.45, and we’ve started to head back towards Uttarkashi. Mallik is really sick, dives his head down and goes to sleep. Just close to Gangotri Rana shows us ‘Pandav gufa’. I wonder where all did Pandav visit during their ‘Karavas’ and ‘Agyatvas’. We find Pandav places from Nasik to Himalayas, 50% of India in length covered !? The ride back now seems tiring with the thought of 5 hrs of travelling back to Uttarkashi on the bad roads. We all doze off, intermittently getting up to check the spots, trying to store our memories.

Pilot Baba Ahram, Maneri

4.30 pm, Sign boards show Maneri and Rana asks us if we want to see Pilot Baba Ashram. We all are ready. The Ashram is a beautiful place loaded with flowers and statues. There are various flowers, roses, sunflowers, lilies and so on. Rajoba is quick to show us the flowers and various other plantations. He gets nostalgic remembering his Sangli garden and how it was destroyed. There are various statues of Indian dieties, Ram, Hanuman, etc. We spend almost an hour when I hear shouts, Rana wants to get back soon and we hop in the jeep and are back heading towards Uttarkashi.

Uttarkashi

We reached Uttarkashi somewhere around 6.15, at Devlok we feel exhausted with the day’s travel not more than 200kms but in those bad roads and the twisting and turning ghats, it was tiring. Mallik needs to go to a Dr., we find a local Dr. Bhandary and take him there. He gives Mallik all drugs for Stomach infection. Around 8.30 we leave Devlok for food, But we are not interested in Bhandary hotel’s meal so try to find alternatives. Luckily we find a juice store, where we jam-in various juices, purchase some Lichi. We come back to the room and start packing our bags for the next day’s long distance jeep travel to Hrishikesh.

Trekking in Satara

Satara is a place of plains showered with small hills all around. At times these hills are in a queue, gives an impression of being placed manually.

Vairatgad

Vairatgad Fort (Wai T; 17° 50′ N, 73° 50′ E; RS. Wathar, 21 m;) in Wai 3,939 feet above sea level, lies nine miles north-east of Medha and six miles south-east of Wai. The base village is Vyajvadi.


Vairatgad photos

Nandgiri / Kalyangad

Nandgiri or Kalyangad Fort (Koreganv T; 18° N, 74° E; RS, Satara Road, 2 m. w;) 3,537 feet above sea level. Nandgiri aka Kalyangad, The hill with a big peepal tree in the dead center at the very top. Its so prominent that Nandgiri and its tree can be spotted from other hills in Satara. It is separated from the rest of the spur by a small gorge or Khind and stands on a lower hill than the Candan Vandan range close to its north-west. mmediately after entering the lower gate a steep footpath descends within the western was into a hollow at the bottom of the scarp. The hollow is about forty or fifty feet deep. When the bottom is reached the entrance to a huge cavern is seen which can only be reached by stooping. The cavern is full of water but very dark. Outside the gate and to the north there evidently were huge caverns but these have been built up with rough masonry.


Nandgiri/Kalyangad photos

Pateshwar

Patesvar (Satara T; RS. Rahimatpur, 9 m, E) a peaked hill rising above the rest of the range about seven miles south-east of Satara, has on its north-west face close to the junction of the villages of Deganv, Nigdi and Bharatganv and within the limits of Deganv a series of cave temples. From Deganv a path strikes to the south-east and winds up to a khind or gorge from which by steps in places it proceeds at a very gentle incline for about three quarters of a mile along the hill side till the temples are reached.


Pateshwar photos

Sajjangad

Parali or Sajjangad [The name Sajjangad that is the fort of good men or sajjan is locally said to be derived from the number of good men who visited it after it became the residence of Ramdas Svami] Fort (Satara T; 17° 35′ N, 73° 00′ E; RS Satara Road 16 m. NE; p. 1,196) about 1045 feet above the plain and 1824 yards in circumference.

The only entrance to the fort is by two gateways at the southwest angle. Both the gateways are in good order strongly built of cut-stone and flanked by towards and a parapet along the rock. The lower gateway which is partly under the rock is completely hidden from the approach and commanded by the upper gateway.


Sajjangad photos

Reference

Dodital Trek and Uttaranchal: Day 7 (14 Jun 08)

This entry is part 8 of 13 in the series Himalayan Trip: Dodital Trek and Uttaranchal

Day 7, 14 Jun 08

Bebra Ghat

My stomach is grumbling, I feel bad, need to go to the loo, I wake up, Its dark, everybody is sleeping. Probably its still night. I need to go. The loo I remember is not in the room, I need to get down walk and there is something built over there. I don’t have time to find a torch, I wake up Vj and aks for a torch. I get down the stairs, and walk towards the toilet scared of leeches jumping on my foot. Its liquid ! I return back and get into the bag, its around 2.00am. I’ve added myself to upset stomach list with Mallik and Vj.

 

6.00 am, Sound sleep in a nice warm rest house always has enemies. Rajo’s alarm, heartless. We somehow manage to fight our sleep and start getting ready. Viki will never start the day without some food in our stomach, ‘Daliya’ is the morning’s breakfast after a request by Mallik. We come outside all geared up for the last day’s walk, the owner of the house has a air-gun that we play around with and finally bid Bebra Ghat a final goodbye around 8.45. Down the hill and up again changing mountains for Agora. At Agora we gave a small halt as Viki returned the mattresses. Agora was yet again shining the early morning sun, with a lovely valley in the front, and itself being on a terrace field, Agora was one of the most beautiful places I’ve seen. There were many flowers that had blossomed and none missed an opportunity to captivate the nature.

Along the way, Viki showed us the Bhang plants, and various flora. At Agora he said, is a split for Bugiya Dayal, which on route towards Gangotri and Valley of Flowers. The Valley of Flowers blossoms best in August. Viki continued on his trek-adventure-talk, that during the peak season of Valley of Flowers the smell is so intoxicating that someone can get dizzy and fall. So to prevent that they keep smelling Garlic and chew it, just to fade of that fragrance of flowers. On our way I saw an Apple tree for the first time in my life. The apples were small and green in color, with just a haze of red pushing through.

My legs were tiring, and it felt I was climbing down with more than 20kgs. I now felt that a little climb could do some good for the legs. But what goes up must come down, so did we, kept descending… Finally around 11.00am we saw the Asi Ganga the rivulet that joins Bhagirathi at Sangamchatti, which meant Sangamchatti wasn’t far enough. 15 minutes later we were there, back in the civilization, roads, motor vehicles for transport, dry shelter, tapri for chai and more. We crossed the road and waited, Soon enough Viki came running, he had arranged for a Commander/Jeep to take us to Uttarkashi. We were moving through the jeep along the Bhagirathi river, for the first time since 5 days, we were travelling and yet not walking or climbing. The Bhagirathi continued to flow, shaping up people’s lives around it.

Uttarkashi

12.30 pm, The town was familiar, It was just a few days since we’d left from here, but felt like a year or more. We waded through the civilized world’s hassles, traffic jams, Bad roads, dirt, etc. We kept looking for an ‘affordable’ hotel/lodge, but didn’t find it easily. Finally we settled for one called ‘Devlok’. Had a ‘clean’ bath after many days, all these days it had been just a fast water flowing, I had a neat shave. And the last bath had been at Bebra Ghat while we were climbing towards Dodital. A neat dry set of clothes and we felt fresh instantly. We sent Nikhil with Viki to arrange for a cab for the next day’s Gangotri visit.

4.30 pm, Most of them are ready and we leave for food. We go to Bhandary hotel and have omelette bread and tea. Mallik has a slight fever. The omelette is wonderful ! Vj didn’t come and is busy washing clothes. We get back to the room with Paneer pulav and Dahi. I went to the top of the hotel, was just a concrete roof, with no boundaries. It was a splendid view from there, the towering hills on the left side with Bhagirathi and the ‘Zula’ or the bridge on it. I waited there for almost 2 hours till Rajoba came and got online with his phone and many others came up and were online.. While just stared in the hills. Viki sincerely called back, We had asked him if he could accompany us to Gangotri the next day, But he said it could be difficult for him, as he has to prepare for another big trek with group of army men.

We came downstairs, and started watching the snaps, it was close to 7.30pm, we didn’t realize how time passed and it was 10.00pm and we yet had to have our dinner. we rushed back to Bhandari hotel and have Dal, Paneer bhurzi and roti. Back at the hotel, we went to sleep on the bed, something I hadn’t done since leaving home. But while we slept we had next day’s alarm already ringing at 3.30am for Gangotri.

Dodital Trek and Uttaranchal: Day 6 (13 Jun 08)

This entry is part 7 of 13 in the series Himalayan Trip: Dodital Trek and Uttaranchal

Day 6, 13 Jun 08

Dodital

6.00 am, Rajo’s alarm is sickening. We get up and find a place for a food-dump. The guard again points that do not do it near the stream that feeds the pool. We find some good place, after that, I was feeling a lot better. Others too finish off, and start doing the ‘packing’ business. With a big backpack and loads of things to carry, the whole packing takes around 5 minutes or more. Viki as usual was up and running, around 8.00 we have our breakfast of Rice and Dal, Viki encourages us to push in as much as we can, as there is a lot to climb and so lot of energy to be consumed. Vijay is feeling nauseatic and drinks two glasses of Salt water, but that doesn’t help him vomit. The strange feeling continues to spoil his morning.

 

9.30 am, We finally leave for Darwatop. That was our first night halt between Dodital and Hanuman-chatti. Within 1/2 km in our route, and the rain plays spoil sport. It had been so sunny all morning, and now !!!! We stop under a big rock, The rains make it worse for trekkers, with the water in the bags, its even more weight to carry. 15 minutes into our first break, and the rain still hasn’t stopped, but worsened. We discuss the plan of a trek in these rains. We decide against the porters’ wishes and continue in the rain. Me, Vj and Mallik are still apprehensive, but Amey is fixed on the plan. So we continue ahead in the hard rains, Its not even a minute after we’ve moved ahead, and Viki finally calls for a stop. This time he orders that we should not move ahead. He continues that It would be too risky ahead in the rains, There is no stop from now right till the Darwatop. So we have no place for a break. The ration too would get spoilt and there would be nothing for food right until Hanumanchatti, So we’d be left with no food and shelter ! .. and we decide to return back. Its raining cats, dogs, elephants, aligators !! We got back to Dodital in 15-20 minutes, I headed straight for the tapari for some tea, while others went to the temple. The porters soon followed, one of them was drenched. He quickly came to the fire and started drying himself. I too followed.

Manjhi

12.30 am, I’m still at the fire, the rain is till bad, Rajo comes to call me and I follow him with my sack to the temple where everyone is busy drying themselves and packing their bags. Soon we had put the foot on the accelerator and heading downwards in the direction of Manjhi. Somewhere around 2.30 its Manjhi, the hazy, chilly Manjhi. The rains had disappeared as we descended. Probably they just didn’t want us to go further ahead. Our stomachs were crying for food, so Maggi was on the menu. Vj got out his wax stove and prepared maggi. While he did that we had some hot tea at Premsingh Ranwat’s tapri, it was the same place where we’d stopped for the night the previous day. There were some kids helping him, on of them was Manisha, she was from Agora village and had come here for vacations. We were discussing whether these kids came here for some work. The kids almost all girls were named like heroines, Priyanka, Rita, Sonam and more. We take snaps of our bags all lined up, our dirty and mucky shoes and leave Manjhi by 3.45pm towards Bebra ghat, which was going to be the night halt.

Bebra Ghat

Coming down from Manjhi we had to change mountains which meant some climbing, our mind was now set on ‘Rest – Go back’ mode. So a little climb was uneasy for us, but we continued through the rainy coniferous forests. Kacheru was on the way by 4.20pm, We discussed the correct pronunciation of the place, Kacheru, Kachori and so on. We halted for few minutes refilled our water bottles and floored the pedal again. Bebra village’s boundary invited us close to 6.00pm. We descended a little further in the village and found a rest-house 2 rooms Rs. 300 a night. Viki was asked to prepare everything tonite, Dal, Roti, Tomato chatni, all that he could manage. We didn’t have any other night halt now, so the ration could be used away.

10.30 pm, No food yet, we were already resting, planning our future tip details, arguing over whether we should stay in Mussoorie or not, etc. We had finally come back to the civilization mode, planning trips in Buses, cars, taxis etc. Viki finally came calling for food, but we were now too lazy to easily jump off our sleeping bags and beds, Yes the hut even had 3 beds ! quite a luxury. We forced Viki to tell some ‘bhoot ki kahaniyan’, something that Viki had himself promised but never implemented. At food, everybody was relaxed, noramlly the porters always seemed strenuous after the days’ porting and thinking about the next day, but today everyone was joking, enjoying. After food we went back to ‘our’ room and laid back. Viki came back again, with his Sherpa smile of his on the face.

आप लोगोंको कहानी सुननी थी ना !

And he narrated two incidents of his, one when he was carrying a cylinder on his back to his home, when the cylinder suddenly swayed and he felt that somebody had put a foot on his slipper. He is still recovering from change in motion when he feels a thorny hand on his back. He turns back to see nobody around. Another one he passed us was about his walk back to his home, which he feels is full of ghosts. He was walking through the ‘Shamshanghat’ when he saw a Bear-Man like creature on the corner, He tries to look properly in the dark but is dazed by the bright red eyes of the creature. He is scared to death and runs back. After that the every curious Vijay keeps asking Viki a zillion questions on ‘Bhoot’, ‘pret-aatma’. Viki finally departs and we fall asleep by 11.00pm.

Dodital Trek and Uttaranchal: Day 5 (12 Jun 08)

This entry is part 6 of 13 in the series Himalayan Trip: Dodital Trek and Uttaranchal

Day 5, 12 Jun 08

Off from Manjhi

7.00 am,
After a treacherous and strenuous trek last day, its going to be a relatively easy walk to Dodital from Manjhi. We don’t get up before 7.00 and slowly gather courage to repack our sacks and muster enough strength to load it on our backs. Even the night’s sleep is not without effort. Every move that I made in the sleeping bag made me pant for breath! Thats how I got through the night. Viki had made Aloo paratha and we had those and left around 11.15. The trek was more of a walk on a slope, sometimes ascending and othertimes just flat. At times when we crossed two mountains, the slope divided towards the base, and we were just happy enough for the descend as we were for more climb. Along the route Viki kept informing us about the various trees, mosses and how even the mosses are loaded full in trucks and sold off. We came across 2-3 bridges on streams which were in construction phase. Viki knew some of the labourers. The streams in Himalayas are endless. In Western India people are desparate for waterfalls and streams in the mountains, over here, its just another natural feature.

 

Dodital

1.45 pm
A board read “Dodital”, But a lake was nowhere to be seen. All I could see was the normal greenery, hills and the valley behind. But a turn and the water body manifested, like a scene from Mahabharata or Ramayana. A lake goddess suddenly appearing in front of its worshippers!

The whole scene was serene and cold. There was no direct sunlight, just diffused light, making the whole place even colder. The ‘Dodital’ lake in Hindi means the lake of Dodi or lake of Ganesha. It was surrounded by small hills on three sides and the other side by the valley from where we had entered the area. The hills were full with Coniferous trees, a small cloud hanging low in one of the hills, another side getting a light drizzle, it was very mystical. With the lawns in bright green color, the water in a dark shade of blue and gray surrounded by dark green tall trees. This is Dodital, the place visitors can only reach by a trek, no vehicles no clumsy tourists, just pure nature lovers are found here.

We decide to roam around, Taking a lot of snaps and videos. there was a small wall around the lake, just to keep people off. We climbed the wall and there was a gaurd making sure that we didn’t get into the lake. Swimming, Angling/Fishing, bathing is not allowed inside the lake. The locals gaurd it like a deity. We circled the lake in an anti-clockwise direction, moving through puddles, small streams that added fresh water to the lake from all the sides. When we reached towards the entrance side of the lake and worshipped in the Ganesha temple we reailized, that people worship first and then make a प्रदक्षिणा around the lake in a clockwise direction.

I and Vj are not entirely feeling good, both of us are nauseatic. We decide we’ll sleep off in one of the huts (an octagonal hut) which was for tourists. I had Pudinhara and we cosily slipped into our sleeping bags. We were woken by the noise Rajo, Mallik and others were making when they had returned from a more adventurous trip around the lake. They had been climbing one of the hills around with a porter and while descending both of them had slipped. Each of them trying to hold eachother for grip and both sliding hands held together, just like from a hindi movie where the hero and the heroine slide on snow, the difference here was no heroine and the snow was actually transformed into pebbles and rocks that just kept hurting the bums.

7.00pm, It was dark and raining heavily, when Viki called us for food. We had Dal and Roti. I was still a little nauseatic but better managed 1 1/2 thick rotis. The rain was just getting worse when we were inside. And after food we had to wait to commute to the other hut where we were planning to sleep. The kitchen or the other hut where we had our food, was to be used by porters and Viki.

8.30pm, We finally managed to get back to ‘our’ hut. But the rains were hard and had taken its toll on the roofs of our huts. With our engg brains and headlights, we were trying to find a dry place. But there was none. Everytime we thought that there was some space and we moved the 3-legged bed, a helpful drop would make our efforts go away. Mallik was even trying to draw a wet line around the dripping hole that would make the water flow to the other side, thanks to something called as surface tension. But the rains were hard and were not to be tamed. We finally decide that we had to put the tents inside the hut to make our night sleep dry and cosy. We had spent more than and hour and a half on this task of finding a dry place. Viki came in and started to arrange the nets. When we were getting into our sleeping bags, we were apprehensive of any leeches in our bags. Around 9.30-10.00 we finally managed to settle and bid adieu to the day, for the next day we had a difficult climb to Darwa top.

MySQL, Memcache, Replication and Delay

An interesting article on Facebook, Which deals with MySQL’s replication system, Memcache and the replication delay.

… by setting a cookie in your browser with the current time whenever you write something to our databases. The load balancer also looks for that cookie and, if it notices that you wrote something within 20 seconds, will unconditionally send you to California. Then when 20 seconds have passed and we’re certain the data has replicated to Virginia, we’ll allow you to go back for safe pages.

The following is just a dump of the URL: http://www.facebook.com/notes.php?id=9445547199

I joined Facebook in April 2007 and, after getting settled over the course of a few weeks, my manager Robert Johnson approached me. We talked for a while but the conversation boiled down to:

Bobby: “So, Jason, we’re going to open a new datacenter in Virginia by 2008. Do you think you can help?”
Me: “Uh…. yes?”
Bobby: “Great!”

My first project at Facebook was a tad more involved then I was expecting, but I think that is one reason why we have such a great engineering organization; we have a lot of hard problems to solve and everyone here is excited to jump in and tackle them. I set out to really understand why we were building a new datacenter and what problems we had to overcome to make it work.

Why Bother?

The primary reason for building a new datacenter on the east coast was latency. It takes about 70 milliseconds to send a packet across the country on a high-speed link, and it can be much longer for an average internet user. By putting servers in Virginia we could reduce the time to send a page to users on the east coast and in Europe by a noticeable amount.

Secondary concerns were space, power, and disaster recovery. We were running out of physical space in our primary datacenters in California and the Virginia site would give us lots of room to grow. We were having a similar problem with getting enough electricity to power all those servers. Finally, restricting ourselves to only one location meant that, in the event of a disaster (power failure, earthquake, Godzilla), Facebook could be unusable for extended periods of time.

Build It!

Before we could go to work on the application level challenges our operations team put in a heroic effort to build out the servers and the physical space in Virginia. They also brought up the intra-datacenter network and the low latency inter-datacenter fiber channel link. This work was an enormous undertaking but our operations team is top-notch and made it all look easy.

With the network and hardware in place we set up our standard 3 tier architecture: web server, memcache server, and MySQL database. The MySQL databses in Virginia were going to run as slaves of the west coast databases, so we spent a couple weeks copying all the data across the country and setting up replication streams.

Now that the hardware, network, and basic infrastructure was set up it was time to face the two main application level challenges: cache consistency and traffic routing.

Cache Consistency

A bit of background on our caching model: when a user modifies a data object our infrastructure will write the new value in to a database and delete the old value from memcache (if it was present). The next time a user requests that data object we pull the result from the database and write it to memcache. Subsequent requests will pull the data from memcache until it expires out of the cache or is deleted by another update.

This setup works really well with only one set of databases because we only delete the value from memcache after the database has confirmed the write of the new value. That way we are guaranteed the next read will get the updated value from the database and put it in to memcache. With a slave database on the east coast, however, the situation got a little tricky.

When we update a west coast master database with some new data there is a replication lag before the new value is properly reflected in the east coast slave database. Normally this replication lag is under a second but in periods of high load it can spike up to 20 seconds.

Now let’s say we delete the value from Virginia memcache tier at the time we update the master database in California. A subsequent read from the slave database in Virginia might see the old value instead of the new one because of replication lag. Then Virginia memcache would be updated with the old (incorrect) value and it would be “trapped” there until another delete. As you can see, in the worst case the Virginia memcache tier would always be one “version” behind of the correct data.

Consider the following example:

1. I update my first name from “Jason” to “Monkey”

2. We write “Monkey” in to the master database in California and delete my first name from memcache in California and Virginia

3. Someone goes to my profile in Virginia

4. We don’t find my first name in memcache so we read from the Virginia slave database and get “Jason” because of replication lag

5. We update Virginia memcache with my first name as “Jason”

6. Replication catches up and we update the slave database with my first name as “Monkey”

7. Someone else goes to my profile in Virginia

8. We find my first name in memcache and return “Jason”

Until I update my first name again or it falls out of cache and we go back to the database, we will show my first name as “Jason” in Virginia and “Monkey” in California. Confusing? You bet. Welcome to the world of distributed systems, where consistency is a really hard problem.

Fortunately, the solution is a lot easier to explain than the problem. We made a small change to MySQL that allows us to tack on extra information in the replication stream that is updating the slave database. We used this feature to append all the data objects that are changing for a given query and then the slave database “sees” these objects and is responsible for deleting the value from cache after it performs the update to the database.

How’d we do it? MySQL uses a lex parser and a yacc grammar to define the structure of a query and then parse it. I’ve simplified the following for ease of explanation, but at the highest level this grammar looks like:

query:
statement END_OF_INPUT {};

statement:
alter
| analyze
| backup
| call
… (insert, replace, select, etc.)

Pretty straightforward, right? A query is a statement which breaks down to one of the MySQL expressions we all know and love. We modified this grammar to allow appending memcache keys to the end of any query, as follows:

query:
statement mc_dirty END_OF_INPUT {};

mc_dirty:
{}
| MEMCACHE_DIRTY mc_key_list;

mc_key_list:
mc_key_list ‘,’ text_string { Lex->mc_key_list.push_back($3); }
| text_string { Lex->mc_key_list.push_back($1); };

A query now has an additional component; after the statement comes the mc_dirty which is either empty or a keyword MEMCACHE_DIRTY followed by a mc_key_list. A mc_key_list is just a comma-separated list of strings and the rule tells the parser to push all the strings one-by-one on to a vector named mc_key_list which is stored inside a per-query parser object.

As an example, an old query might look like:
REPLACE INTO profile (`first_name`) VALUES (‘Monkey’) WHERE `user_id`=’jsobel’
and under the new grammar it would change to:
REPLACE INTO profile (`first_name`) VALUES (‘Monkey’) WHERE `user_id`=’jsobel’ MEMCACHE_DIRTY ‘jsobel:first_name’

The new query is telling MySQL that, in addition to changing my first name to Monkey, it also needs to dirty a corresponding memcache key. This is easily implemented. Since the per-query parser object now stores all memcache keys we tack on to a query, we added a small piece of code at the end of mysql_execute_command that dirties those keys if the query is successful. Voila, we’ve hijacked the MySQL replication stream for our own purpose: cache consistency.

The new workflow becomes (changed items in bold):

1. I update my first name from “Jason” to “Monkey”

2. We write “Monkey” in to the master database in California and delete my first name from memcache in California but not Virginia

3. Someone goes to my profile in Virginia

4. We find my first name in memcache and return “Jason”

5. Replication catches up and we update the slave database with my first name as “Monkey.” We also delete my first name from Virginia memcache because that cache object showed up in the replication stream

6. Someone else goes to my profile in Virginia

7. We don’t find my first name in memcache so we read from the slave and get “Monkey”

Page Routing

The other main problem we had to address was that only our master databases in California could accept write operations. This fact meant we needed to avoid serving pages that did database writes from Virginia because each one would have to cross the country to our master databases in California. Fortunately, our most frequently accessed pages (home page, profiles, photo pages) don’t do any writes under normal operation. The problem thus boiled down to, when a user makes a request for a page, how do we decide if it is “safe” to send to Virginia or if it must be routed to California?

This question turned out to have a relatively straightforward answer. One of the first servers a user request to Facebook hits is called a load balancer; this machine’s primary responsibility is picking a web server to handle the request but it also serves a number of other purposes: protecting against denial of service attacks and multiplexing user connections to name a few. This load balancer has the capability to run in Layer 7 mode where it can examine the URI a user is requesting and make routing decisions based on that information. This feature meant it was easy to tell the load balancer about our “safe” pages and it could decide whether to send the request to Virginia or California based on the page name and the user’s location.

There is another wrinkle to this problem, however. Let’s say you go to editprofile.php to change your hometown. This page isn’t marked as safe so it gets routed to California and you make the change. Then you go to view your profile and, since it is a safe page, we send you to Virginia. Because of the replication lag we mentioned earlier, however, you might not see the change you just made! This experience is very confusing for a user and also leads to double posting. We got around this concern by setting a cookie in your browser with the current time whenever you write something to our databases. The load balancer also looks for that cookie and, if it notices that you wrote something within 20 seconds, will unconditionally send you to California. Then when 20 seconds have passed and we’re certain the data has replicated to Virginia, we’ll allow you to go back for safe pages.

Looking Back

Nine months after our first user viewed a page in the Virginia datacenter we’re still running the same architecture with good success. There were bumps along the way, of course; for the first month or two the cache consistency infrastructure was very shaky and would periodically force us to divert traffic away from Virginia while we diagnosed and fixed bugs. Over time, however, we’ve ironed out the issues and now serve a substantial portion of Facebook’s traffic out of this datacenter.

The main scaling challenge with this architecture is pretty obvious: all write operations must happen in one location. Going forward we’re very excited to develop new technologies that will let us perform writes in any location. We’re also thinking a lot about how to use our new datacenter as a disaster recovery site in case Godzilla decides to attack our California locations! Interested in helping us out? www.facebook.com/jobs!

Adding Fedora 9 DVD as repository

Adding a Fedora 9 DVD as a yum repository is extremely easy in Fedora 9.

For slow internet connections, using yum could be pain, as most of the software can be installed from the DVD itself. But yum resolves dependencies from the net. Adding a Fedora 9 DVD as a repository is like a boon to such users.

Follow the steps using a root user

  1. Pop in your Fedora 9 DVD, It should be automatically mounted on /media/Fedora 9 i386 DVD/ folder
  2. Create a new file fedora-dvd.repo in /etc/yum.repos.d/ folder.
  3. Put the following contents in the file

    [fedora-dvd]
    name=Fedora 9 DVD
    baseurl=file:///media/Fedora%209%20i386%20DVD/
    enabled=1
    
  4. Disable fedora.repo and fedora-updates.repo, by assigning the parameter enabled=0.
  5. Run the command yum repolist
    You should be able to see fedora-dvd as enabled.

Thats should be it. You are free to install packages from the DVD directly.

ऋतुराज का Home Page