From espians

The Team Of Total Awesomenesses Shaila

Info: See <http://openideaproject.org/irclogs/browse/esp-core/latest> for introductory docs.

Author: Jeffry Archambeault (jeffarch) <jjarchambeault@gmail.com>

Date: $Date: 2006-06-17 21:00:00 +0100 (Sat, 17 Jun 2006) $

Revision: $Revision: 0001 $

Description: Summary of daily #esp-core IRC logs

Shaila Meta-Stuff

References

_devplan: http://www.espians.com/espra/devplan.html document covers the details.
_devtasks: http://www.largeblue.com/_space/first_run_through.txt

#esp-core Summary

[Business Contracts]

[Business Structure]

[Business Accounting]

Discussed hardware costs, storage & bandwidth charges

- GReaper trying to figure out licensing
- tav £2k/mo for corporate clients and £1.4k/mo for non-corporate
- tav on top they get charged for bandwidth and any customisations
- GReaper costs i mean :p
- tav ah, software licensing ?
- tav basically, we need to pay for quicktime, windows, flix, and an unknown buffer for any others
- GReaper ah, flix license is a one off when you buy more servers
- tav requests to encode use the premium bandwidth
- tav yeah, the licenses increase as we add new servers
- tav new windows servers to be specific
- tav we have a 2 servers for each "encoding" -- one linux to do the preparation work (e.g. pal <-> ntsc, hard subtitling) and hopefully flash, and one windows to do most of the other formats
- tav need to negotiate down that flix license obviously
- tav but, even at initial price, we're profitable

There's a spreadsheet discussed at http://openideaproject.org/irclogs/browse/esp-core/2006/06/16#15-26-54

[Development Timeline]

[Development Tasks]

- oierw tav: so is there code I can look at yet?
- tav oierw: not yet -- gonna talk to sbp about structuring
- sbp no code until tav talks to me about structuring
- sbp that's the rule 

...

- tav atm, i've got 2 separate directory structures -- kalati and protoplex
- tav should i merge both together ?
- sbp protoplex is the old code that actually works, right?
- tav kalati's trunk/source/python is designed for pimp
- sbp and kalati is the new code, bits of which are used by protoplex
- sbp does kalati work?
- tav only protoplex is used in green.tv -- in proto.util package, there are some modules copied over from kalati's pimp structure
- tav proto being the python package for protoplex
- tav sorry for the confusion caused by the varying mix of the names
- sbp no problem, as long as it's stable now :-)
- tav heh, yea
- sbp I suppose you only need to throw the bits of protoplex into kalati that kalati is currently missing, and that kalati needs to make it work
- tav right
- tav only slight problem is perhaps that protoplex and kalati treat trunk/home differently ?  

[Infrastructure]

- GReaper temp.espnow.net has address 72.21.40.139
- GReaper plan of action
- GReaper today/saturday/early-sunday, move qin's essential services to temp.espnow.net
- GReaper sunday afternoon, install slackware/xen, qin becomes host1.espnow.net
- GReaper sunday evening, virtual servers on host1.espnow.net
- GReaper services can then move back to that box

...

- tav have you and evangineer had a chance to discuss our general server infrastructure for butterfly.tv ?
- GReaper nope :p

Big, deep discussion here...

- GReaper i just see a load of servers split up so early as a waste
- tav hmz
- tav GReaper: a waste in what aspects?
- GReaper awkward to setup/develop/run
- tav why ?
- GReaper hmmmmm
- tav trying to understand what difference it makes if server 4 is in the usa or japan?
- GReaper if it's in the same rack, it's cheaper :p
- GReaper lemme see what i can find
- tav ... ?
- GReaper getting a bit pissed off at the ones which don't even give a price
- tav heh
- GReaper "click here to get a quote!"
- GReaper you know what sounds tempting?
- GReaper loads of those hetzner 4200 dual core athlons, 2GB ram, 2x160GB disks, 3GB bandwidth
- GReaper £60 per month each
- tav right
- GReaper seems odd
- tav what does?
- GReaper http://www.hetzner.de/flatrate.html
- GReaper they offer 10mbit flatrate for 199 EUR
- GReaper and 100mbit for 1999 EUR
- tav it's more expensive though ?
- tav i thought 10mbit == 3TB ?
- GReaper exactly
- GReaper :p
- GReaper 10 of those servers = 790 EUR
- GReaper 100mbit is 1999 EUR
- tav --
- tav 01:59 <   GReaper> awkward to setup/develop/run
- tav --
- tav does that still apply?
- GReaper yup :p
- tav your main point was financial not technical... ?
- GReaper the awkwardness bit you mean?
- tav *nod*
- GReaper technically awkward
- GReaper you need a *complete* copy of all data at all locations
- tav oh
- tav that'll be taken care of by the storage layer -- i.e. by the application code
- tav it's a key development requirement
- GReaper okay, next awkward bit
- GReaper where do the requests go to?
- tav i was hoping we could work that out ;p
- tav dunno much about advances in load balancers and such
- tav can we put up "frontends" on a round robin dns which then delegate appropriately ?
- GReaper that's one possibility, a frontend at each location
- GReaper however, you'd have to find a provider which allows free data transfer between servers
- GReaper right? :p
- tav as long as they are on the same switch most seem amicable towards that
- tav unless you mean between the different locations ?
- GReaper i meant at the same location
- GReaper quite a lot just charge at the port
- tav well, hetzner are cool with that
- tav we can make sure that the others we go with are too
- GReaper others aren't :p
- tav then we won't go with them ;p
- tav will vary tbh. from 10 gigs to terabytes. source video files are HUGE
- GReaper so, maybe some servers attached to a SAN would be more appropriate?
- tav how cheap are SAN solutions ?
- GReaper http://www.coraid.com/purchase.html
- GReaper £2300 for the thingy
- GReaper £4200 for the drives
- GReaper £6500 or so for 11250GB of disk space
- tav http://labs.google.com/papers/gfs.html
- GReaper okay
- tav thoughts ?
- GReaper just take, say, hetzners servers as an example
- GReaper 2x160GB drives
- tav k
- GReaper you'd need 36 servers just to get the same amount of storage space
- GReaper ( = £2160/month )
- tav but you get the benefit of the processing and the i/o ?
- GReaper for what?
- GReaper this isn't google :p
- tav hmz, the indexes, the queries, &c.
- tav well, yeah, it's meant to be better than google ;p
- tav talonlzr also pointed out earlier that we might want to keep a lot of data in ram
- GReaper hmmm?
- tav e.g. indexes and high demand content
- GReaper you can query from the SAN and cache in RAM
- tav good idea
- tav so, what exactly does the SAN offer in comparison to say machines with several large hard drives ?
- tav guess it's RAID that i have a problem against and not SAN in general ?
- GReaper less storage space per unit of rackspace
- GReaper that thing fits in 3U, 36 servers is probably 72U
- tav can't we fit bigger hard drives into our servers ?
- GReaper 3 drives a server is still 10U
- GReaper + the actual cost of 5 servers
- tav *nod*
- tav makes sense
- tav and presumably SANs can still handle heavy I/O when RAID isn't being used ?
- GReaper urm
- GReaper you'd still use RAID? :p
- tav hmz, RAID provides redundancy which we don't want we do it on the application layer?
- tav s/want/want as/
- GReaper why do you want application redundency?
- tav because that's one of the key features of the plexnet ;p
- tav being a distributed network and all that  - GReaper tav tav tav
- GReaper you are doing media streaming, right?
- tav nope
- GReaper wtf is it all about then?
- tav straight http downloads
- tav heh
- GReaper media streaming then
- tav same effect less the streaming servers, yes
- tav plays in your browser and with the exception of shitty wmv format even seeks before the download is complete
- GReaper so
- GReaper if you have say, a SAN, connected to 4 webservers
- GReaper why would you even NEED application redundency?
- jeffarch idea would be for each user to be part of the storage
- jeffarch that would help tie users to reputation / plexchange / pecus
- tav ehm, jumping ahead a few steps there jeffarch ;p
- jeffarch ;p
- tav GReaper: so server Europe-12 has file A. suddenly file A becomes popular and file A gets cached along various other servers in our network.
- tav the idea is not to have specific instances tied to specific servers
- tav but, instead, to just the data float around within our various servers
- GReaper really varies on what you're trying to offer
- GReaper a media streaming service, or a fancy application network
- tav GReaper: hmz, initially focussing on providing "tv on internet" (+ community) -- but the idea is to extend that to a fully featured collaborative media platform which can run all  sorts of funky applications using the distributed indexes and storage ;p
- tav oierw: well, we could put up a linen sheet
- oierw and I agree with GReaper. The point right now is to develop a distributed media system, not to try to get our servers to run the plex on the application level when there are  simplier methods to do what we want... well... i kind of agree
- oierw a series of small steps will get us further. a large step may put us into the ravine
- oierw s/will/may
- tav ravine how?
- tav developing a functioning dht isn't rocket science and the benefits are tremendous
- tav ^ oierw 
- jeffarch the machines have large enough drives for several clients, with some exceptions... so is the current issue of backups?
- jeffarch like use the SAN to backup all of the clients?
- tav jeffarch: that way lies pain ?
- jeffarch 6 weeks worth?
- tav we can build a good dht in 6 weeks
- GReaper or you can use an existing one
- tav which will provide us with a scalable, distributed system so that we don't have to worry when a server dies or hardware fails
- tav GReaper: that too ;p
- GReaper you still have to worry
- tav i want us to be in the position where we can simply and cheaply add capacity as needed without having to care if one of them dies
- tav GReaper: true
- oierw` if this is a democratic process, I recommend we do not write our own dht at this time
- GReaper so, £6500 for 10.5GB of usable space isn't cheap enough? :p
- GReaper erm
- GReaper 10.5TB
- GReaper £135/month approx as a 2 year investment
- tav GReaper: that bit is cheap, but monitoring/administrating/fixing them isn't?
- tav unless we use the various RAID options, at which point they stop being cheap too
- tav oierw: reasons ?
- oierw` tav: the fewer things we build, the faster we have a product that has the promised features
- oierw` we could fork the protoplex into working and dev trees and work the dht seperatably (since somewhere down the road we are actually going to need one, i'm sure)
- tav oierw`: hmz
- GReaper it isn't cheap to maintain them
- oierw` well, we probably are going to want a custom dht for katali or the protoplex
- oierw` but to get everything working, it makes sense to use a dht with very few modifications. Right now it's just between our servers
- GReaper but it's probably a lot cheaper than a shitload of servers elsewhere
- oierw` i think the two of you are arguing different visions. what do you forsee the solution you are discussing ending as?
-  tav "ending as" == ?
- tav as espra
- tav butterfly.tv is the first step towards espra
- GReaper urm
- oierw` GReaper is seems to want to get something up that can handle the huge demands we will place on it right now. tav seems to be looking towards the future
- oierw` (now being the next 6-12 months)
- oierw` but I could be wrong :)
- GReaper on the spreadsheet, you've got january 2007, 52 linux servers
- GReaper where the hell you expecting to get those from?
- tav ?
- tav whichever hosts we decide upon
- tav hetzner seems to be a decent choice so far; need to find similar others
- GReaper assuming the same disk space as hetzner (240GB)
- GReaper you'd get 12480GB of space
- GReaper i'd say this needs to go in several stages
- GReaper stage #1, host at hetzner on an additional server or two
- GReaper cheap, effective
- GReaper stage #2, get your own servers and colocate them somewhere, get a SAN
- GReaper stage #3, then you distribute out and get racks in other datacentres
- GReaper does that not sound more logical? :p
- tav re: stage2 -- hiring out something like hetzner helps our cashflow better?
- oierw does it hurt to have two datacenters that are independant of eachother (allowing natural disasters and such)
- oierw it maxes out there. your rate rate is negative => your growth per month should start going down
- GReaper hiring out me perhaps to manage servers in a datacentre when needed? :p
- tav GReaper: hmz, we're budgeting to have you and evangineer on full time to take care of all these servers ?
- GReaper i know
- GReaper but the point is, renting out a few servers to start off with fine
- GReaper it's cheap, it gets you started
- GReaper however, theres a point at where your own servers in your own rack is far superior
- tav oierw: *nod*
- GReaper and cheaper
- oierw GReaper: how long would it take to go from a single datacentre to multiple datacenters?
- tav GReaper: agreed
- oierw assuming we are trying to design the system with that in mind
- GReaper say, 1-2 years?
- tav guys we've told clients that we'll have multiple datacenters...
- tav well, we mentioned 2
- tav usa and europe
- GReaper easy enough
- GReaper okay, i'll think about it a bit more-

...

- GReaper for example, hetzner give you a full rack, with 100mbit connection for £1500

...

In my (jeffarch) defense, my "jumping ahead a few steps" was part of a thought-chain regarding DHTs and free drive space on each server.  If each client had their own server, there'd seem to be lots of extra drive space available for 3x redundancy, via DHT.  2 SANs (eu and us) plus the host server would also give 3x redundancy, but I don't know how DHT's figure into that.

[Development Team Members]

[Protoplex/Kalati]

[VOIP]

[Licensing]

[Subversion]

svn & track installed

- tav there's an xslt stylesheet that can be applied to the svn btw
- tav http://svnbook.red-bean.com/en/1.0/ch06s04.html#svn-ch-6-sect-4.5.1

- tav http://divmod.org/svn/Divmod/sandbox/exarkun/commit-bot/ looks useful

- jeffarch we still want the tags-plugin for trac, right?
- tav oh yea

[LOCALIZATION]

[Transparency]

[3ware]

[butterfly.tv]

[Development Summary]

[World Cup]

[Other]

==TTLY== - This Time Last Year - a look at the #esp logs from a year ago, just for fun

mattis tries Gobby for ting

Deep discussion between Sunir and qopi about wikis

- Omni|Work Someone had signoff message: "Never underestimate the power of stupid people in large groups!"
- Omni|Work And I figure I should make one: "Never underestimate the power of cycnicism to promote apathy."