Someone asked me what actually was cloud computing, and I found it rather difficult to answer concisely and coherently.  Much arm waving was involved, and comments like “you know, like Amazon”, and “you get billed for your usage”.  I thought it was worth looking at a clear, simple definition!

What is cloud computing?

Cloud computing is

“a computing capability that provides an abstraction between the computing resource and its underlying technical architecture (e.g., servers, storage, networks), enabling convenient, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction.”

according to the National Institute of Standards and technology.  Thats fine, and the more technical among us could probably leave it at that.  How would you try describing cloud computing to your CEO, though?

Cloud computing can be thought of as a way of pooling the resources of a number of computers together, and then pouring those resources into whatever applications you decide to run on the top of them in a very simple way, allowing you to set up new applications very quickly and easily.  Instead of having to build a new server for a new application, you can pour some more resource from the cloud for it.  If the resources in the cloud are running low, just add some more servers to the pool.

In the past, there have been a variety of mechanisms that do parts of this, but haven’t applied it all together. 

Older, similar technologies

Clustering computers has been done for some time.  This allows computers running a application to work together, providing redundancy for the application, and generally some performance improvement.  How does this differ from a cloud?  Well, a cluster was generally quite complex to set up, and only really focussed on a single application.  You couldn’t pour resources from all the machines into a new application quickly at all.

Web farms can support many web applications, running over lots of web servers.  How does this differ from a cloud?  Well, web farms again tended to be hard to configure, and it was very difficult to apply specific levels of resources to specific web sites – if one web application used lots of processing power, the others could start to run more slowly.  Administration and maintenance to balance these problems, and to roll out new applications were often complex.

Virtual server providers are almost like mini clouds, where running virtual machines are provisioned simply and easily across the available hardware.  How do they differ from cloud computing?  Well, cloud computing separates off storage and processing power, allowing you to apply more storage or more processing power as needed to any application running on the infrastructure.  It adds another level of abstraction, which allows much more effective (and quick and easy) application of the necessary storage space and processing power from all the resources pooled in the cloud, rather than limiting it to a single virtual server, which can then be quiite complex to change.

All of these technologies really combine to provide a simple infrastructure on top of a pool of servers – you have the performance and resilence of a cluster, without the headaches of setting it up, and without the limitations of the focus on a single application.  You have the distibution of a web farm, allowing multiple applications to share resources, but with much easier tools to support rolling out new applications or providing additional resources to underperforming ones.  You have the benefits of virtual servers, but with better applications of storage space and processing power to your actual applications.

Confusing technologies and terms

There are two points that many people don’t quite understand about cloud computing, even if they grasp how it can be applied.  These are:

Does a cloud have to be on the Internet?

No!  A cloud can be run inside a private corporation, or even at home if you have the resources.   Ubuntu provide cloud deployment and management tools as part of their open source enterprise servers.   The reason cloud computing is synonomous with the internet in people’s minds is that the real benefits of cloud computing require massive, massive deployments of hardware to see the real benefits and cost savings.  Having said that, even a small cloud might be better for supporting a slew of internal web applications than 15 individual servers.

Because of the costs of a large cloud, only companies like Amazon have successfully deployed them as yet.  They use clouds to sell processing power and storage space to any one who wants to buy them.

Will I be charged by processing use or storage space?

This is actually a separate concept, called Utility computing, where you are charged for your system use like you are charged for gas, or electricity, hence the name.  This is by no means integral to the concept of cloud computing – though Amazon EC2, the biggest cloud deployment, does use this model to charge for access.  It doesn’t mean every has to charge this way, and an internal corporate cloud would just cost electrical consumption, hardware and licensing.

Don’t confuse the two – utility computing is a model of charging for computer use and storage space, while cloud computing is an infrastructure for pooling computer resources to be poured simply into applications on demand.  They may both be used at the same time, but aren’t the same thing.

Setting the hostname on an Ubuntu linux installation is pretty easy during the installation.  However, for a standard image to be dropped onto a wide range of machines, it really needs to be unique and set via a script.

We like to use the serial number of the PC in question, though this is more or less practical depending on the brand of PC.  Acer, for example, tend to have VERY long serial numbers – it may be unique, but its going to be harder to type than just an IP address.  IBM tend to use reasonable 7 digit strings, which is is much more usable.  Other brands can be longer or shorter.

Setting hostname itself seems easy – just run

hostname <newname>

(where sudo is used to run the command as root) in a terminal, and the hostname of the machine will change.  You’d think we were done, but unfortunately not!  This will only change the hostname until the next reboot.  On startup, the contents of the file

/etc/hostname

is used to set the  hostname.  To update this, I like to set the hostname for the current sessions, then run

hostname > /etc/hostname

(as root) which willoverwrite the file with the current hostname.  If running as part of a script on startup, this will set the hostname now, then update the file for future restarts.  However, we still aren’t done.  We need to update another file

/etc/hosts

with the details of the name for networking purposes.  We need to add the hostname, and any domain name aliases too.  The hosts file will probably look something like:

127.0.0.1                        localhost                 localhost.domain.local

127.0.1.1                        <hostname>

Update the hostname line, and add new aliases for the hostname for any domains that may be relevant.  At this point, the system is renamed!  However, this is all still pretty manual – ideally we need to script the process.

Now, to get the serial number, we can query the bios using the dmidecode command, and then process it usign the myriad of linux text handling commands.

dmidecode | grep “Serial Number” | head -n1 | sed -e ‘s/\tSerial Number: //g’

(as root, again) should return the serial number from the bios!  We can combine this with the hostname command, as follows:

hostname $(dmidecode | grep “Serial Number” | head -n1 | sed -e ‘s/\tSerial Number: //g’)

(once again, as root.) This will set the hostname to the serial number.

I actually combine all of the commands discussed to form a single script -

hostname $(dmidecode | grep “Serial Number” | head -n1 | sed -e ‘s/\tSerial Number: //g’)

hostname > /etc/hostname

echo “127.0.0.1       localhost     localhost.domain.local” > /etc/hosts.new

echo “127.0.1.1     ”$hostname”     “$hostname”.domain.local” >> /etc/hosts.new

mv /etc/hosts.new .etc/hosts

This sets the hostname, then updates the hostname file.  It then generates a complete hosts file line by line, and overwrites the old version.  There are probably better ways of updating the text file directly, but this works effectively enough.

Finally, I set this script file (which I named hostname.sh) to run on system startup.  Simply copy the file to

/etc/init.d

and run (as root)

update-rc.d hostname.sh defaults

where hostname.sh is the chosen name for your script.  This will add the script to the startup scripts on the machine, where it will automatically be run as root.

Recently, I’ve been playing with Linux, specifically Ubuntu, in an attempt to set up a simple, maintainable client for virtual desktops.  Its been a fair while since I’ve used linux in a serious sense, so I thought I’d post up what I’ve done, as I progress (largely for my own reference, but hopefully others might find it of use!)

Key requirements are:

A virtual client!  In this instance, the vmware open client will need to be installed and configured on the desktop.  There are still limitations with the open client that may break the plan – limitations with remote media playback, and usb redirection are two areas in particular that may cause issues.

A working web browser!  Of course, Firefox is an obvious standard, installed with Ubuntu, so thats not much of a challenge, at least on the surface.  Beyond a working web browser, we need to possibly extend our server architecture to support browsers beyond Internet Explorer for our key wep applications, allowing a level of work to be carried out in the event of virtual desktop failures.  This is where things get a lot harder!

A standard environment across different hardware, locked down for the default user.   This is actually quite tricky.  By default, linux is designed to be easy to customise and configure, so locking it down to a single user, while allowing network proxy changes and wireless connections, is actually quite a challenge.  In addition, desktop launchers will need to be variable, depending on local printer installations for users on laptops with home printer (vmware-view allows you to redirect a printer, but you need to specifically do it by name).

An architecture to allow remote reconfiguration, support and updates across a company wide platform.  This is one of the worst areas – linux still lags quite badly behind the sort of architecture taken for granted on a Windows network when it comes to administration through global policies.  It’s still fundamentally a server operating system,and admin tools generally focus on supporting machines runnign in that capacity, not as clients.  This is the hardest area of all, looking forward to a possible roll out of well over a thousand machines world wide, with a technical team with no linux experience!  Keeping the client simple and cheap, allowing machines to be swapped instead of supported is a very high priority where possible.

Hopefully the next series of posts on this topic will be useful, although its quite a change of tack from SharePoint!  Don’t worry, as new SharePoint issues come up, I’ll still be posting on that topic too.

Tom commented on a post with the following problem, and I thought it merited a post.

“We have a MOSS 2007 FARM AND 3 DOMAINS all have a two way trust.  We have over 78 sites all of which stopped with no known reason from being able to find users that are in one of the domains we can find the users in the view profiles yet we can no longer find users using peoplepicker for any users from the one domain.

We have tried this command you have provided and they come back with commandline error
stsadm -o setproperty -pn peoplepicker-searchadforests -pv domain:full domain name,-userlogin domain\username password –url http://webapp url”

It sounds like an interesting problem, but its difficult to answer without more information.  Incidentally, the profiles comment, about being able to view user profiles, is rather a red herring.  This is handled by an import process specified elsewhere in the SSP, and has nothing to do with the People Picker displaying users.

Lets discuss the stsadm command first.   Without knowing the specific error message, I can’t say why the command is failing, but there are two probable outcomes.   stsadm is not generally included in the default path for a windows installation, so if the error message is:

‘stsadm’ is not recognized as an internal or external command, operable program or batch file.

The problem is simply that you need to find the appropriate location first.  The location is generally:

C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\BIN>

so simple change directory to that path, and run the STSADM again.  It sounds obvious, but I must admit, I had a bit of a nightmare trying to find the stsadm path when I first started looking at SharePoint!

The other possibility is that the command has been run, but that the parameters haven’t been entered correctly.  If so, you get the terribly helpful response of :

Command line error.

Followed by a complete syntax reference for the command, and it sounds much more likely that this is the cause of the problem, from the notes in the question.  Unfortunately, this is quite a lot harder to discuss, as unsuprisingly people aren’t going to give the specific command line with all of their configuration details and passwords to be put onto a website, and the generic, censored versions are probably going to be correct, at least as far as it goes.

The best help I can give here is to put together a full hypothetical example, rather than just repeating the command syntax yet again.

Essentially, you first need to set an internal SharePoint encryption key, then tell the server what domains to add to the list, and what valid username and password to use to connect to the domain in order to pull back the list.  Don’t use administrator, btw!!!

To set the initial encryption key, use:

stsadm.exe -o setapppassword -password <yourencryptionkey>

To set the actual domain link, use:

stsadm.exe -o setproperty -url http://domain1.example.com:80 -pn “peoplepicker-searchadforests” -pv “domain:domain1.example.com,domain1\LoginName, P@ssword; domain:domain2.example.com,domain2\LoginName, P@ssword; domain:domain3.example.com,domain3\LoginName, P@ssword“

Of course, with two way trusts a single user name and password could be used if you granted the appropriate rights. 

What normally goes wrong putting this together?  Normally it is either the encryption key hasn’t been set first, or that construction of the domain list has a syntax issue (or that the surrounding quotes have been missed off).

As a rule, though, if you get a command line error when running stsadm, you have got the format wrong.  If the format is right, it won’t necessarily solve the problem (if your username or password is wrong, for example, it still can’t access the other domain information), but you’ll see the changes applied.  A good way of checking is to run:

stsadm.exe -o getproperty -url http://domain1.example.com:80 -pn “peoplepicker-searchadforests”

And it will show the details you’ve set (with passwords asterisked out).

What makes the problem Tom is experiencing interesting is that apparently the People Picker has been working, and now isn’t.  This, to me, implies that something has changed with the installation, or the Active Directory configuration.

Key things to check would be:

Has the system user context still got access rights to the domain that appears to be no longer accessible?  A two way trust means that user rights can be assigned … by default they arent.

Have the active directory servers changed?  If so, you may need to force DNS updates, otherwise resolution against the AD may be looking for defunct servers.

Has the stsadm command been run succesfully in the past, and the usernames and passwords have since changed (or expired)?  This will obviouslly drop off the people pickers ability to query the domain.

Has a security patch been applied against the domain, or has permissions to the Active Directory been changed?  By default, older systems allowed anyone to do a basic LDAP query against the Active Directory, but this was locked down.  If this loophole was previously being used instead of setting correct access rights for the security context of the wep application, it’ll obviously start causing this issue.

I hope this helps – Tom, if you’ve solved the problem, please let me know the solution

OK, before I get started here, let me clarify.  I’m predominantly a SharePoint administrator, not a developer, so these reviews are based around their use for installation and administration, as opposed to their use for a web developer!  If I think something would be useful for different roles, I’ll try and point it out, but do bear the viewpoint in mind as you read further.

With the disclaimer out of the way, lets get started with O’Reilly’s “Essential SharePoint 2007″ (You can find this at Amazon.co.uk and Amazon.com).  I’ve got mixed feelings about this book.  I used it a lot when getting started with SharePoint, and its a great reference when I go back to do something that I’m not doing frequently.  However, it lacks depth, and seems a bit unfocussed, covering areas that developers, administrators and users all need to know.  Almost the entire team I work with, both developers and admins, have a copy of the book and use it on occasion, but its not a great place to start, and not the best place to go for a detailed technical reference either.  I’d say its absolutely ideal for someone tackling SharePoint in a smaller company, where you need to have an understanding across the board, rather than specialising in a particular area.  In a larger company, its a great book if you occasionally work with SharePoint and want a solid technical reminder.

Surprisingly, one of the most useful books for me for SharePoint was  “Microsoft SharePoint 2007 for Dummies” (You can find this at Amazon.co.uk and Amazon.com).  This may be because I’m a dummy, or just my method of approaching a new technology.  I read through a simple introduction to the technology, to get a rough idea of how it works, what the components are, how they fit together, and what the terminology is.  Because I start with a simple, clear and concise book , I can pick up that information really quickly, then I can use that information to bootstrap myself up to the complex technical specifics, and actually get the most of the in depth technical references.  The “Microsoft SharePoint 2007 for Dummies” is perfect for this, providing a really solid introduction to the concepts and terminology.  I think it’d also be useful for a developer trying to understand how SharePoint fits together and is likely to be used, and very useful for a technical manager who just needs broad brushstrokes while his team does the detail work.

I wasn’t keen on the “Microsoft Office SharePoint Server 2007 Administrator’s Companion” (You can find this at Amazon.co.uk and Amazon.com).    Others may feel differently about it – many of the technical people I work with love the style in these Microsoft Press books.  I find that its too focussed on how to achieve specific goals by clicking specific buttons, rather than focussing on the particularly settings you need, and why they are needed.  Its more a philosophical difference than anything – I’m not just focussed on results, I need to know why I’m doing something.  I’ve found a solid understanding of a technology leads to much better results … and a much better ability to troubleshoot issues … than simply knowing what you need to click to achieve a particular goal.  Unfortunately, that’s the path Microsoft in general have taken with their books and courses, and increasingly seems to be the approach taken by the technical people I meet.  If thats what you want, this book is perfect.  I didn’t get on with it.  If you check the Amazon reviews, however, you’ll find most people love it.

“SharePoint 2007 The Definitive Guide” (You can find this at Amazon.co.uk and Amazon.com) was pretty good as a guide.  Some aspects are very good – its coverage of SharePoint installations and upgrades from 2003 is excellent, for example.  Its coverage of network topologies and security is also excellent.  Where this falls down, and where the Internet shines, is the fact that I’ve found you need to know how a range of technologies work together to cover many of the odder demands of a SharePoint installation.  If you publish a SharePoint site, you’ll probably want to use ISA Server 2006 to secure it, not just rely on SharePoint.  Using a SQL server on a different domain via SQL authentication isn’t uncommon for DMZ deployments.  This book is great for vanilla SharePoint installs, and is definitive for basic admin tasks, such as deploying and configuring basic sites, roles and permissions.  You’ll need to look elsewhere for any installs beyond the vanilla.  I rarely go back to this book, but I’d thoroughly recommend reading through it at least once, and using it as a reference if you’re actively looking after SharePoint sites, as opposed to looking after the architecture.  If you need to get to grips at level of the stsadm command line, you won’t find it here.

Publishing SharePoint via ISA Server is fairly straightforward, even if you use SSL to the reverse proxy, and connect on port 80 to the SharePoint server in order to move the encryption load from your SharePoint box.

However, for some reason, this appears to break Telerik’s RadEditor.  In order to use RadEditor, you need to be able to connect directly to the server in the same format of the initial request, which means in the case of a secured SharePoint site, SSL.   Simply change the bridging rule to redirect requests to the SSL port, and disable redirection to the HTTP port.  This will place more load on your SharePoint servers – every request is encrypted via SSL between the SharePoint server and the proxy – weigh this load up against the editing benefits of RadEditor.

One of the common problems I’ve seen with Microsoft SharePoint is the loss of search functionality, and I’ve found a lot of different theories and possible solutions.  I’ve tried to combine the various ideas into a step by step troubleshooting strategy.  Pleae note that this only applies if you’ve had a working earch service in the past – if not, you need to enable the search service before you do anything else!

Symptoms

Individual sites and site collections respond to any search query with no results.

The event log on the SharePoint Server shows Event  ID 2436 – The start address <site url> cannot be crawled.

The crawl log within SharePoint says that the site cannot be crawled and has been deleted from the gatherer.

Possible Resolutions

First, disable loopback checking on IIS, if you haven’t already.  This is one of the most common causes of the problem, and if it isn’t done, means that you WILL experience the problem sooner or later anyway.  To disable loopback checking, you need to make a registry change and restart your server, so schedule it for a quiet time.

Follow these steps:

1. Click Start, click Run, type regedit, and then click OK.
2. In Registry Editor, locate and then click the following registry key:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
3. Right-click Lsa, point to New, and then click DWORD Value.
4. Type DisableLoopbackCheck, and then press ENTER.
5. Right-click DisableLoopbackCheck, and then click Modify.
6. In the Value data box, type 1, and then click OK.
7. Quit Registry Editor, and then restart your computer.

Second, check the permissions on the search service.  It sounds silly, but it is easy to sometimes use an account that doesn’t have access rights to actually crawl the sites!  Don’t get too involved with this, however – if it looks right, it probably is.  Its easy to waste days checking odd possibilities of access rights, but if this is the cause, its normally pretty obvious!

Third, ensure that there is a site at the top level of the web application.  This sounds ludicrous, but I’ve seen systems spring into life after a blank site is deployed at the root of the website.  Its easy to check, and easy to fix.  Many people won’t have seen this, as it’s pretty common practise to deploy self service site creation in the root url.  I’ve particularly seen this on systems after an SP2 MOSS and WSS install.  If a 404 error is received on the root url, all the other sites won’t be crawled.

The SharePoint usage reports, once enabled, look extremely pretty, but are generally pretty meaningless unless you are dealing with absolutely huge numbers of hits, where averages and graphical representations are the only effective way of dealing with information.  However, there are two report pages that are extremely useful, particularly for slightly smaller sites, that can’t be reached through the GUI interface in MOSS 2007.   They are actually from the basic WSS system, and MOSS inexplicably misses out any direct reference though the administration pages.

The second of these, usage details, is brilliant for tracking usage of a site, showing which users have accessed the site on which days over the last month.  Its invaluable as a quick security check to make sure no unexpected users are accessing the site!  The permissions system in SharePoint is pretty robust, especially if you’ve stuck to SharePoint and AD groups to assign rights, but its always possible for oddities to happen.

Enabling usage reports on SharePoint should be straightforward – after all, basic information about the use of a Site or Site Collection is pretty important.  Of course, it isn’t as straightforward as you would think.

First, you need to enable “Usage Analysis Processing” in the Operations -> Usage Analysis section of the Central Administration site.   Enable Logging has to be ticked, in order to have information to process.  Make sure you have space in the logging folder – and make sure you set the permissions right if you change the default directory:

C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Logs

You also need to enable usage analysis processing, and set a time to process the logs.  Particularly with larger logs, this can be quite intensive, so you’ll need to schedule it to occur well outside peak usage hours.  There doesn’t seem to be any option to process it immediately when testing, so you’ll have to wait for the data to appear over a day or so.

In most guides, this is where the process stops.  However, you need to go into the Shared Service Provider, and under Office SharePoint Usage Reporting – Usage Reporting, you’ll find another set of options which must be enabled.

Ensure that both “Enable advanced usage analysis processing” and “enable search quey logging” are both ticked.  At this point, the system is live for usage reports … though you’ll be waiting for data until the scheduled time!

Having read various papers on this over the course of the day (the most useful can be found here), it seems pretty clear – the SharePoint security structure is hard to maintain!

Here are some key recommendations for a maintainable structure:

• Focus on using Active Directory to maintain your security structure.  Set up groups in Active Directory, and assign them to SharePoint groups in your site collection.  Don’t assign users to SharePoint groups – assign users to the right groups in Active Directory, which will automatically filter through into SharePoint.
• Active Directory has a wide selection of APIs and tools to manage group membership – if you need to handle security changes programatically, you’ll find it much easier in AD than SharePoint.
• Always use Domain Local groups in Active Directory to assign to your SharePoint group.  This is not only best practise according to standard Windows recommendations, it also allows you to use users from a different domain over a one way trust.  Universal and Global groups are limited to your domain and forest respectively.
• SharePoint’s People Picker is quite powerful.  If you need granularity beyond simple groups, you can assign specific access to individual users where needed, but your administration demands will go up.  Try not to break inheritence rules without VERY strong business drivers, as maintaining security after that point is extremely difficult.
• For a simple extranet structure, using the default SharePoint groups is fine – most external users are visitors to the site, unable to make changes.  Internal users are members, able to update teh site where appropriate.  IT staff tend to be Site Owners, in order to maintain the site structure.  This is actually quite flexible, especially if moving between the groups is simple through Active Directory group assignments.