Month: May 2012

Categories
General

Using Open Source to Provide Infrastructure Services

 
 
From: http://blog.moovweb.com/2011/10/using-open-source-to-provide-infrastructure-services/

Operations Teams need to provide eight critical services to the developers and users of their environment.  At my current employer, I use open source software to provide these services that allow our developers to be more productive and our customers to experience stable, responsive service.

Source Code Management
Keep all of our bespoke software, configurations and notes under strict version control.

Sofware: Git
Pros: Fast
Stable
Many developers familiar with it due to Github’s popularity
Cons: Steep learning curve
Somewhat cryptic commands
Option: Subversion

 
Continuous Integration
Build, test, version and package our software so that it may be quickly and safely deployed to our staging environment

Sofware: Jenkins
Pros: Easy Integration with GIT
Nice GUI
Flexible enough to meet our needs
Cons: Configuration limited to GUI
Written in Java*
Option: Cruise Control

 
Provisioning
Spin up nodes to become part of the processing farm and decommission nodes no longer required

Sofware: Custom scripts using Fog
Pros: Simple scripts
Easy to customize
Support multiple cloud providers
Cons: Custom tool
Option: Cobbler, RightAWS

 
Configuration Management
Ensure that all nodes are automatically and correctly configured and remain in a known configured state

Sofware: Puppet
Pros: Easy configuration language
Well supported
Active community
Cons: Have to learn said configuration language
Requires serious investment of time
Option: Chef

 
Monitoring
Check on services and nodes to ensure that things are behaving as expected before the customer notices
 

Sofware: Icinga
Pros: Can be easily auto-configured by Puppet
Well understood
Nagios syntax
Works well with nagios checks and plugins
Cons: Requires serious investment of time and constant care
Option: Nagios, Zenoss

 
Capacity/Performance Management
Collect system metrics for assessing performance and capacity planning.  Some organizations have monitoring perform this role, but I have very strong opinions on this being kept separate.
 

Sofware: Collectd/Visage
Pros: Light, fast daemon on each box
Flexible server
Many plugins availble
Cons: Separate process to run
Requires a lot of disk and disk I/O
Option: Ganglia

 
Log Collection
Centrally collect, store and monitor system and application logs

Sofware: Rsyslog/Graylog2
Pros: Rsyslog provides flexible configs
MongoDB backed server performs well
Easy front end for log viewing
Cons: Takes a while to learn Mongo
Harder to pull/backup then text logfiles
Options: Syslog-ng Logstash

 
Deployment Management
Allow developers and technical staff to deploy and monitor application activity.  Since each infrastructure is unique, it makes sense to build a custom solution to this problem.
 

Software: Mcollective/Sinatra/ActiveMQ
Pros: Sinatra makes it easy to write simple web applications
Mcollective is extremely fast
ActiveMQ is very flexible and resilient
Cons: Sinatra is not a full featured as Rails
Mcollective requires a change of thinking about command/control
ActiveMQ is Java*
Options: Control Tier

 
* I list Java as a con because we do not have extensive in-house Java expertise and it rquires us to install something we would not have normally

Categories
General

Links about the Future of System Administration

 

Categories
General Interoperability OpenSource

Interoperability in the Cloud

Many people in the industry believe it is critically important for the Cloud to be open and share concerns about the private nature of the leading Cloud platforms. In fact, there are already a few projects focused on the goal of a truly open source Cloud with mass adoption.
However the present Cloud offers have followed this trend and are largely private. No one benefits from a fractured landscape of closed and incompatible Clouds where migration is difficult to do and true Cloud transparency is impossible.
Nowadays the solution to interoperate between Clouds is using Web Services. For example, Amazon EC2 has a web service interface to manage his own virtual-machines, VMware has a vCloud interface and other products have their own web services interface. But the problem is that these interfaces are private and based on their own Clouds Computing. Eucalyptus Systems considers API AWS (Amazon Web Services) the default standard for the industry because of its popularity. Eucalyptus Cloud is an open-source virtualization middleware but it uses an Amazon EC2 interface. On the other hand, OpenNebula propose an open source interface called OCCI very easy to use and to extend. OCCI was originally initiated by UCM (Complutense University of Madrid) and now the Open Cloud Computing interface comprises a set of open community-lead specifications delivered through the Open Grid Forum.
Usually the APIs wars have been a crucial strategic plan to control the technology platforms and their associated markets. I don’t know if δ-cloud or OCCI will be an API reference of the next years. But I have a clear idea. An open standard API should emerge. Now both have very good fundamentals and are present in many discussion forums. In conclusion we chose to use API OCCI.
We would like to highlight that our OCCI API is used in a lot of research and production projects as , OpenNebula,  EMOTIVE, OpenStack,  Fed-Cloud, MeghaCloud
I believe that the next step is creating another standard working group that sits on top of them all. 
A Cloud Admin

Categories
Social

Chef or Puppet + IaaS = No More Sysadmins?

The DevOps Zone is presented by DZone with partners including ThoughtWorks Studios and UrbanCode to bring you the most interesting and relevant content on the DevOps movement.  See today’s top DevOps content and be sure to check out ThoughtWorks Studio’s Continuous Delivery Whitepapers and UrbanCode’s Webinars.

The end of days is nigh for the profession of systems administration according to Julian Dunn, a digital media systems designer and architect.  At a recent meeting of the New York City Linux Users’ Group, a presentation using the configuration management tool, Chef, led him to the conclusion that

IaaS, or “cloud computing”, now allows anyone to provision new (virtual) servers inexpensively. No more waiting around for the system administrator to order a couple servers from Dell, wait a few weeks for them to arrive, rack them up, configure them, etc. Developers, armed with a tool like Chef and its huge cookbook of canned recipes for making many standard infrastructure components, can fire up everything they need to support their application themselves. Therein lies the demise of system administration as a standalone profession and the rise of “devops”. —Julian Dunn

Traditional CM tools have been composed of elaborate prescriptive, rather than descriptive, scripts that are unaware of the underlying semantic meaning of user requests, according to Dunn.  The tools that we’re now associating with DevOps (Puppet, Chef, etc.) allow you to describe your infrastructure in what Dunn refers to as a ‘4th Generation Language’ way.  That simplified process for creating reusable, canned recipies for configuration, paired with IaaS (where you don’t need to worry about the physical setup and configuration), is what could make the Sysadmin, as we know it, obsolete.  This is Dunn’s theory.
With these new opportunities in cloud computing, Dunn sees the benefits of bringing development and system administration together, and suggests that “if sysadmins want to remain relevant, they’ll get on board and start learning a bit more about programming.”  Good point, Dunn, but let’s not forget that, even with cloud services, code ultimately runs on servers/disk/memory. The hardware is never truly “virtual”.
Source:  http://server.dzone.com/articles/cm-tools-and-end-systems
Memfis (Cloud Admin)

Categories
In detail

Opensource Hypervisors: XEN or KVM

Xen is a hypervisor that supports x86, x86_64, Itanium, and ARM architectures, and can run Linux, Windows, Solaris, and BSDs as guests on their supported CPU architectures. Xen can do full virtualization on systems that support virtualization extensions, but can also work as a hypervisor on machines that do not have the virtualization extensions: For example, Atom and ARM (that are some interesting low power processors) and older CPUs. Also if you want to run a Xen host, you need to have a supported kernel.
KVM is a hypervisor that is in the mainline Linux kernel. Your host OS has to be Linux, obviously, but it supports Linux, Windows, Solaris, and BSD guests. It runs on x86 and x86-64 systems with hardware supporting virtualization extensions. This means that KVM is not an option on older CPUs made before the virtualization extensions were developed, and it rules out newer CPUs (like Intel’s Atom CPUs) that do not include virtualization extensions. For the most part, that is not a problem for data centers that tend to replace hardware every few years anyway , but it means that KVM is not an option on some of the niche systems like the SM10000 that are trying to utilize Atom CPUs in the data center.
Xen is running on quite a lot of servers, from low-cost Virtual Private Server providers like Linode to big boys like Amazon with EC2. Xen has been around a bit longer, also it has had more time to mature than KVM. You’ll find some features in Xen that haven’t yet appeared in KVM, though the KVM project has a lengthy to do list but KVM is going to become more prominent in the future. Also, RedHat and Canonical have begun supporting KVM. Also, KVM is not mature project but KVM performance is improving day by day and it is growing in the kernel Linux because KVM is part of the main Linux kernel.
A difficult decision:
Xen and KVM consume little overhead and power. It is hard to choose a winner because it depends of the environment and each one. KVM is rapidly improving through Xen has better management tools than KVM and Xen migrations are more robust.

Categories
General

[Spanish] Cocinando con Chef por Luis Bosque (@luisico)


http://youtu.be/rSIw0pwyfAY
 
En esta charla Luis Bosque (@luisico) nos mostrará lo necesario para empezar a ensuciarse las manos en serio con Chef [1].
Será una charla más o menos introductoria pero que intentará cubrir todos los elementos importantes de esta tecnoloǵia y llena de ejemplos para ver claramente cómo encaja cada pieza en el sistema.
Los temas que se intentarán cubrir son los siguientes:
– Arquitectura de una plataforma con Chef (server, nodes, clients, ..)
– Autenticación
– Recetas, atributos y roles
– Diferencias entre server y solo
– Anatomía de una ejecución de chef-client
– Gestión de la plataforma con Knife
– Extendiendo los recursos de chef
– Diferentes entornos de chef
Esta charla es una de las convocatorias mensuales que organiza el grupo de Madrid Devops (@madrid_devops). En esta ocasión Tuenti se ha ofrecido a alojar el evento en sus oficinas de Madrid y grabar las presentaciones con audio. Si estás interesado en estas reuniones únete a la lista de correo [2] o visita nuestro wiki [3].
[1] http://www.opscode.com/chef/
[2] https://groups.google.com/group/madrid-devops
[3] https://madrid-devops.jottit.com/
Fecha: Jueves 22 de Marzo a las 19:45 – 20h
Duración: Alrededor de dos horas
Lugar: Oficinas de Tuenti
 

Categories
General

EEDC Seminars- Iaas and Mobile Apps

One week after the first seminar, we attended to the second day of the EEDC Seminars, the second day was focused in Iaas and Mobile Apps.
In this second day, they should talk about Iaas, Saas and Mobile Apps, but finally the seminar about Saas was cancelled. So, the presentations were two:

Xavier Guarda and Joaquin Custodio have developed different iPhone applications, like Mountain Angel, that is a GPS tracking, heart rate monitor, and it sends a SMS to the person you want if you fall down with your bike.
He talked about Parse, it is a backend to create projects for mobile apps, it’s a very good choice to start a mobile app and easy, you can make prototypes without deploy anything on servers.

 

  • Infraestructure as a Service by David Suárez (Nexica)

Nexica offers Iaas and hosting services and they have two datacenters, one in Barcelona and another one in Madrid, member of Catnix and Espanix, the Catalan and Spanish Neutral Internet Exchange points. Nexica has a PUE of 1,4 in their datacenters.
David Suárez started talking about cloud categories, if you see cloud by deployment categories it can be Public, Private or Hybrid and if you see by service type it can be Iaas, Saas and Paas.
Iaas, Saas, Paas and visibility
He talked about community cloud, that are services with specific requirements, restricted to a group of organizations.
David ended his seminar talking the next step in this evolution: infrastructure as code, Iac is an abstraction of infrastructure into configuration files and code. Management and automation soft is needed (puppet, chef, juju). In many cases this evolution is changing the system administrators and devops tasks.

 
This was the last EEDC seminars. Here you can find all the slides of the EEDC seminars.
Enjoy it!
A Cloud Admin