.htaccess tutorial

htaccess Elite


robots.txt usage and examples for robots

Search Engine Optimization using Htaccess

robots.txt usage and examples for robots

Postby ti89 » 28 Nov 2006 02:15

Table of Contents
  1. Abstract
  2. Introduction
  3. Specification
  4. Access method
  5. File Format Description
  6. The User-agent line
  7. The Allow and Disallow lines
  8. Formal Syntax
  9. Expiration
  10. Examples
  11. Backwards Compatibility
  12. Interoperability
  13. Security Considerations
  14. References




Abstract

This memo defines a method for administrators of sites on the World- Wide Web to give instructions to visiting Web robots, most importantly what areas of the site are to be avoided.

This document provides a more rigid specification of the Standard for Robots Exclusion, which is currently in wide-spread use by the Web community since 1994.




Introduction

Web Robots (also called "Wanderers" or "Spiders") are Web client programs that automatically traverse the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.

Note that "recursively" here doesn't limit the definition to any specific traversal algorithm; even if a robot applies some heuristic to the selection and order of documents to visit and spaces out requests over a long space of time, it qualifies to be called a robot.

Robots are often used for maintenance and indexing purposes, by people other than the administrators of the site being visited. In some cases such visits may have undesirable effects which the administrators would like to prevent, such as indexing of an unannounced site, traversal of parts of the site which require vast resources of the server, recursive traversal of an infinite URL space, etc.

The technique specified in this memo allows Web site administrators to indicate to visiting robots which parts of the site should be avoided. It is solely up to the visiting robot to consult this information and act accordingly. Blocking parts of the Web site regardless of a robot's compliance with this method are outside the scope of this memo.




The Specification

This memo specifies a format for encoding instructions to visiting robots, and specifies an access method to retrieve these instructions. Robots must retrieve these instructions before visiting other URLs on the site, and use the instructions to determine if other URLs on the site can be accessed.



Access method

The instructions must be accessible via HTTP from the site that the instructions are to be applied to, as a resource of Internet Media Type "text/plain" under a standard relative path on the server: "/robots.txt".

For convenience we will refer to this resource as the "/robots.txt file", though the resource need in fact not originate from a file- system.

Some examples of URLs for sites and URLs for corresponding "/robots.txt" sites:

Code: Select all
http://www.foo.com/welcome.html http://www.foo.com/robots.txt

http://www.bar.com:8001/        http://www.bar.com:8001/robots.txt

If the server response indicates Success (HTTP 2xx Status Code,) the robot must read the content, parse it, and follow any instructions applicable to that robot.

If the server response indicates the resource does not exist (HTTP Status Code 404), the robot can assume no instructions are available, and that access to the site is not restricted by /robots.txt.


Specific behaviors for other server responses are not required by this specification, though the following behaviours are recommended:

  • On server response indicating access restrictions (HTTP Status Code 401 or 403) a robot should regard access to the site completely restricted.
  • On the request attempt resulted in temporary failure a robot should defer visits to the site until such time as the resource can be retrieved.
  • On server response indicating Redirection (HTTP Status Code 3XX) a robot should follow the redirects until a resource can be found.




File Format Description

The instructions are encoded as a formatted plain text object, described here. A complete BNF-like description of the syntax of this format is given in section 3.3.

The format logically consists of a non-empty set or records, separated by blank lines. The records consist of a set of lines of the form:
Code: Select all
<Field> ":" <value>


In this memo we refer to lines with a Field "foo" as "foo lines".

The record starts with one or more User-agent lines, specifying which robots the record applies to, followed by "Disallow" and "Allow" instructions to that robot. For example:
Code: Select all
User-agent: webcrawler
User-agent: infoseek
Allow:    /tmp/ok.html
Disallow: /tmp
Disallow: /user/foo


Lines with Fields not explicitly specified by this specification may occur in the /robots.txt, allowing for future extension of the format. Consult the BNF for restrictions on the syntax of such extensions. Note specifically that for backwards compatibility with robots implementing earlier versions of this specification, breaking of lines is not allowed.


Comments are allowed anywhere in the file, and consist of optional whitespace, followed by a comment character '#' followed by the comment, terminated by the end-of-line.





The User-agent line

Name tokens are used to allow robots to identify themselves via a simple product token. Name tokens should be short and to the point. The name token a robot chooses for itself should be sent as part of the HTTP User-agent header, and must be well documented.

These name tokens are used in User-agent lines in /robots.txt to identify to which specific robots the record applies. The robot must obey the first record in /robots.txt that contains a User- Agent line whose value contains the name token of the robot as a substring. The name comparisons are case-insensitive. If no such record exists, it should obey the first record with a User-agent line with a "*" value, if present. If no record satisfied either condition, or no records are present at all, access is unlimited.

The name comparisons are case-insensitive.

For example, a fictional company FigTree Search Services who names their robot "Fig Tree", send HTTP requests like:
Code: Select all
GET / HTTP/1.0
User-agent: FigTree/0.1 Robot libwww-perl/5.04

might scan the "/robots.txt" file for records with:
Code: Select all
User-agent: figtree





The Allow and Disallow lines

These lines indicate whether accessing a URL that matches the corresponding path is allowed or disallowed. Note that these instructions apply to any HTTP method on a URL.

To evaluate if access to a URL is allowed, a robot must attempt to match the paths in Allow and Disallow lines against the URL, in the order they occur in the record. The first match found is used. If no match is found, the default assumption is that the URL is allowed.

The /robots.txt URL is always allowed, and must not appear in the Allow/Disallow rules.

The matching process compares every octet in the path portion of the URL and the path from the record. If a %xx encoded octet is encountered it is unencoded prior to comparison, unless it is the "/" character, which has special meaning in a path. The match evaluates positively if and only if the end of the path from the record is reached before a difference in octets is encountered.

This table illustrates some examples:
Code: Select all
Record Path        URL path         Matches
/tmp               /tmp               yes
/tmp               /tmp.html          yes
/tmp               /tmp/a.html        yes
/tmp/              /tmp               no
/tmp/              /tmp/              yes
/tmp/              /tmp/a.html        yes

/a%3cd.html        /a%3cd.html        yes
/a%3Cd.html        /a%3cd.html        yes
/a%3cd.html        /a%3Cd.html        yes
/a%3Cd.html        /a%3Cd.html        yes

/a%2fb.html        /a%2fb.html        yes
/a%2fb.html        /a/b.html          no
/a/b.html          /a%2fb.html        no
/a/b.html          /a/b.html          yes

/%7ejoe/index.html /~joe/index.html   yes
/~joe/index.html   /%7Ejoe/index.html yes



Formal Syntax

This is a BNF-like description, using the conventions of RFC 822 [5], except that "|" is used to designate alternatives. Briefly, literals are quoted with "", parentheses "(" and ")" are used to group elements, optional elements are enclosed in [brackets], and elements may be preceded with <n>* to designate n or more repetitions of the following element; n defaults to 0.

Code: Select all
robotstxt    = *blankcomment
| *blankcomment record *( 1*commentblank 1*record )
*blankcomment
blankcomment = 1*(blank | commentline)
commentblank = *commentline blank *(blankcomment)
blank        = *space CRLF
CRLF         = CR LF
record       = *commentline agentline *(commentline | agentline)
1*ruleline *(commentline | ruleline)


agentline    = "User-agent:" *space agent  [comment] CRLF
ruleline     = (disallowline | allowline | extension)
disallowline = "Disallow" ":" *space path [comment] CRLF
allowline    = "Allow" ":" *space rpath [comment] CRLF
extension    = token : *space value [comment] CRLF
value        = <any CHAR except CR or LF or "#">

commentline  = comment CRLF
comment      = *blank "#" anychar
space        = 1*(SP | HT)
rpath        = "/" path
agent        = token
anychar      = <any CHAR except CR or LF>
CHAR         = <any US-ASCII character (octets 0 - 127)>
CTL          = <any US-ASCII control character
(octets 0 - 31) and DEL (127)>
CR           = <US-ASCII CR, carriage return (13)>
LF           = <US-ASCII LF, linefeed (10)>
SP           = <US-ASCII SP, space (32)>
HT           = <US-ASCII HT, horizontal-tab (9)>

The syntax for "token" is taken from RFC 1945 [2], reproduced here for convenience:

Code: Select all
token        = 1*<any CHAR except CTLs or tspecials>

tspecials    = "(" | ")" | "<" | ">" | "@"
| "," | ";" | ":" | "\" | <">
| "/" | "[" | "]" | "?" | "="
| "{" | "}" | SP | HT

The syntax for "path" is defined in RFC 1808 [6], reproduced here for convenience:
Code: Select all
path        = fsegment *( "/" segment )
fsegment    = 1*pchar
segment     =  *pchar

pchar       = uchar | ":" | "@" | "&" | "="
uchar       = unreserved | escape
unreserved  = alpha | digit | safe | extra

escape      = "%" hex hex
hex         = digit | "A" | "B" | "C" | "D" | "E" | "F" |
"a" | "b" | "c" | "d" | "e" | "f"

alpha       = lowalpha | hialpha

lowalpha    = "a" | "b" | "c" | "d" | "e" | "f" | "g" | "h" | "i" |
"j" | "k" | "l" | "m" | "n" | "o" | "p" | "q" | "r" |
"s" | "t" | "u" | "v" | "w" | "x" | "y" | "z"
hialpha     = "A" | "B" | "C" | "D" | "E" | "F" | "G" | "H" | "I" |
"J" | "K" | "L" | "M" | "N" | "O" | "P" | "Q" | "R" |
"S" | "T" | "U" | "V" | "W" | "X" | "Y" | "Z"

digit       = "0" | "1" | "2" | "3" | "4" | "5" | "6" | "7" |
"8" | "9"

safe        = "$" | "-" | "_" | "." | "+"
extra       = "!" | "*" | "'" | "(" | ")" | ","




Expiration

Robots should cache /robots.txt files, but if they do they must periodically verify the cached copy is fresh before using its contents.

Standard HTTP cache-control mechanisms can be used by both origin server and robots to influence the caching of the /robots.txt file. Specifically robots should take note of Expires header set by the origin server.

If no cache-control directives are present robots should default to an expiry of 7 days.




Examples

This section contains an example of how a /robots.txt may be used.

A fictional site may have the following URLs:

  • http://www.fict.org/
  • http://www.fict.org/index.html
  • http://www.fict.org/robots.txt
  • http://www.fict.org/server.html
  • http://www.fict.org/services/fast.html
  • http://www.fict.org/services/slow.html
  • http://www.fict.org/orgo.gif
  • http://www.fict.org/org/about.html
  • http://www.fict.org/org/plans.html
  • http://www.fict.org/%7Ejim/jim.html
  • http://www.fict.org/%7Emak/mak.html

The site may in the /robots.txt have specific rules for robots that send a HTTP User-agent "UnhipBot/0.1", "WebCrawler/3.0", and "Excite/1.0", and a set of default rules:
Code: Select all
# /robots.txt for http://www.fict.org/
# comments to webmaster@fict.org

User-agent: unhipbot
Disallow: /

User-agent: webcrawler

User-agent: excite
Disallow:

User-agent: *
Disallow: /org/plans.html
Allow: /org/
Allow: /serv
Allow: /~mak
Disallow: /

The following matrix shows which robots are allowed to access URLs:
Code: Select all
                                           unhipbot webcrawler other
                                                    & excite
http://www.fict.org/                         No       Yes       No
http://www.fict.org/index.html               No       Yes       No
http://www.fict.org/robots.txt               Yes      Yes       Yes
http://www.fict.org/server.html              No       Yes       Yes
http://www.fict.org/services/fast.html       No       Yes       Yes
http://www.fict.org/services/slow.html       No       Yes       Yes
http://www.fict.org/orgo.gif                 No       Yes       No
http://www.fict.org/org/about.html           No       Yes       Yes
http://www.fict.org/org/plans.html           No       Yes       No
http://www.fict.org/%7Ejim/jim.html          No       Yes       No
http://www.fict.org/%7Emak/mak.html          No       Yes       Yes





Backwards Compatibility

Previous versions of this specification didn't provide the Allow line. The introduction of the Allow line causes robots to behave slightly differently under either specification:

If a /robots.txt contains an Allow which overrides a later occurring Disallow, a robot ignoring Allow lines will not retrieve those parts. This is considered acceptable because there is no requirement for a robot to access URLs it is allowed to retrieve, and it is safe, in that no URLs a Web site administrator wants to Disallow are be allowed. It is expected this may in fact encourage robots to upgrade compliance to the specification in this memo.



Interoperability

Implementors should pay particular attention to the robustness in parsing of the /robots.txt file. Web site administrators who are not aware of the /robots.txt mechanisms often notice repeated failing request for it in their log files, and react by putting up pages asking "What are you looking for?".

As the majority of /robots.txt files are created with platform- specific text editors, robots should be liberal in accepting files with different end-of-line conventions, specifically CR and LF in addition to CRLF.




Security Considerations

There are a few risks in the method described here, which may affect either origin server or robot.

Web site administrators must realise this method is voluntary, and is not sufficient to guarantee some robots will not visit restricted parts of the URL space. Failure to use proper authentication or other restriction may result in exposure of restricted information. It even possible that the occurence of paths in the /robots.txt file may expose the existence of resources not otherwise linked to on the site, which may aid people guessing for URLs.

Robots need to be aware that the amount of resources spent on dealing with the /robots.txt is a function of the file contents, which is not under the control of the robot. For example, the contents may be larger in size than the robot can deal with. To prevent denial-of- service attacks, robots are therefore encouraged to place limits on the resources spent on processing of /robots.txt.

The /robots.txt directives are retrieved and applied in separate, possible unauthenticated HTTP transactions, and it is possible that one server can impersonate another or otherwise intercept a /robots.txt, and provide a robot with false information. This specification does not preclude authentication and encryption from being employed to increase security.




References

  1. Koster, M., "A Standard for Robot Exclusion", June 1994.
  2. Berners-Lee, T., Fielding, R., and Frystyk, H., "Hypertext Transfer Protocol -- HTTP/1.0." RFC 1945, MIT/LCS, May 1996.
  3. Postel, J., "Media Type Registration Procedure." RFC 1590, USC/ISI, March 1994.
  4. Berners-Lee, T., Masinter, L., and M. McCahill, "Uniform Resource Locators (URL)", RFC 1738, CERN, Xerox PARC, University of Minnesota, December 1994.
  5. Crocker, D., "Standard for the Format of ARPA Internet Text Messages", STD 11, RFC 822, UDEL, August 1982.
  6. Fielding, R., "Relative Uniform Resource Locators", RFC 1808, UC Irvine, June 1995.
ti89
 
Posts: 44
Joined: 28 Oct 2006 07:37

Postby ti89 » 30 Jan 2007 12:51

[code]#
#
# (C) Copy and Copyright 2006 WebmasterWorld Inc. All Rights Reserved.
# http://www.webmasterworld.com/robots.txt
# This code found here: http://www.webmasterworld.com/robots.txt?view=producecode
#
# Please, we do not allow nonauthorized robots. They are a major drag on the system.
# Actual robots.txt can always be found here for: http://www.webmasterworld.com/robots2
# Old full robots.txt can be found here: http://www.webmasterworld.com/robots3
#
# Any unauthorized bot running will result in IP's being banned.
# Agent spoofing is considered a bot - if it looks like a bot and not from an SE - it is a bot.
# Honey pots are - and have been - running. If your access has been blocked for bot running - please sticky an admin for a reinclusion request.
#
# http://www.searchengineworld.com/robots/
#

User-agent: *
Disallow: /





# [:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:] #
# #
# ---------------------------=*> THE BOT BLOG <*=-------------------------- #
# #
# [:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:][:] #



# =========================================================================== #
# #
# Waiting to Exhale #
# #
# by brett tabke #
# 9/1/2006 #
# #
# =========================================================================== #
#
# re: http://www.webmasterworld.com/forum10/9410.htm
#
# --------------------------------------------------
# Smart money said that Alan Meckler was going to
# do for Search, what Comdex did for the DotCom and
# what CES did for home electronics.
# --------------------------------------------------
#
#
# August 2nd, 2005. Was a watershed day in the history of SEO/SEM.
# That was the day Jupiter sold SearchEngineWatch and the SES
# conferences to Incisive Media. After the events of this week with
# Danny Sullivan announcing his resignation from SearchEngineWatch, I
# think it is worth rolling back the clock to that fateful day and
# looking at the sale of the site and conferences again.
#
# http://www.incisivemedia.com/
# http://www.daggle.com/
# http://www.SearcheEgineWatch.com
# http://weblogs.jupitermedia.com/meckler/
#
# Search was sky rocketing a year ago. Google and the search sectors
# growth was spectacular. Adsense and other forms of advertising were
# setting new records every month. From SEO/SEM firms to mom-n-pops,
# we not only had survived the dot com crash - we came out stronger
# and more profitable because of it. Everyone was making out like
# bandits. At the time of the sale, Jupiter was doing great. Sullivan had
# the conference side of the conferences running like a well oiled
# machine and the trade show side was also jamming. SES was growing
# by 20-50% per-conference. The site was also rocking and hitting every
# note a webmaster should be hitting today. There are/were 20+ ads on
# the home page alone going for a rumored $4-5k a month - talk about
# your million dollar homepage! Life was good at Jupiter and certainly
# in the search space.
#
# http://finance.google.com/finance?q=GOOG
#
# Jupiter didn't appear to be hurting. Sure the stock price was flat,
# but not crashing by any means. The outlook for Jupiter was fairly bright
# and positive. Alan Mecklers fixation with the images seemed to be
# mostly harmless. Who knows, he may have been on to - still might be -
# something really big with all that recurring billing. In a few old
# school circles, soft licensing and recurring billing are the holy grail,
# come-to-jesus, show-me-the-money of business models. Seriously, guys
# like Bill Gates did all right by soft licensing (grin).
#
# With Jupiter hitting on all cylinders, I just think that at the time
# of the SES sale, Meckler should have been standing back screaming at
# the troops to saddle up. It was a time to redouble their efforts. It
# was setup already. It was a time to hitch up the team and go to town
# on a rail.
#
# Business maven Tom Peters used to say that the time to sell like
# hell" is when "you are already selling like hell"! You have to
# strike the iron when the metal is hot. Nothing was hotter than
# "Search" was last year.
#
# http://www.tompeters.com/
#
# The selling of SES at that time, just did not make sense. Everyone
# I talked to about it was bewildered. Why sell SES? It was a
# stunning and shocking move to those of us in the industry. No one
# could wrap their brain around it. Daddy Warbucks - dude - buddy -
# what the? Go round the bend!? Was Meckler getting ready to retire?
# Has he ticked off the major sponsors? Has he been put on notice? Is
# there some balloon payment due in the backwaters of Jupiters books
# that no one knows about? Contracts due to expire and no renewals in
# site? Someone calling in a loan? Does he think there is another
# bubble crash imminent? Missed a margin call? A greedy ex wife? Like
# the ponies too much? Ha! you could go crazy spinning the scenarios.
#
# http://weblogs.jupitermedia.com/meckler/
#
# Alan Meckler is a seasoned business pro. There are few in the tech
# space that have his years of real world seat-of-the-pants first hand
# old fashioned education you can only get from decades of on-the-job
# training. Aside from that sentence turning a record number of
# cliches per column inch, it's true! Meckler is that old and
# experienced in this space. His name alone is almost a cliche!
# Alan built the awesome Internet World trade show and conference and
# he built SES. You don't fall off a turnip truck and wake up with
# domains like Internet.com, ClickZ, and SearchEngineWatch. With Alan
# at the helm of Jupiter, we were getting that old deja vu Internet
# World feeling all over again. It just stood to reason that Alan Meckler
# was going to do for Search, what Comdex did for the DotCom and what CES
# did for home electronics.
#
# For our part, we were just trying to stay out of the way - to keep
# from ending up road kill. Jupiter needed a wide berth. We mulled
# over thoughts of getting on the Jupiter wagon. We even met with the
# big guy himself (thanks for the lunch!). There was no kidding
# ourselves about our future direction. We were just caught in the
# draft of it all. We were along for the ride whether we wanted to or
# not - sucked in to the wake of the SES/SEW vortex. Fates and
# fortunes are intertwined. So goes Jupiter - so goes WebmasterWorld.
# It is the pure definition of that new catch word from competition
# and cooperation: co-opetition. There were days I could see my
# handwriting on SES/SEW, and I am sure there are days Danny saw his
# on PubCon/WebmasterWorld. And why not!? We are all doing well -
# making money - having fun - cranking out 7 figure daily traffic
# numbers - traveling the world - partying like rock stars - three
# finger snaps in a Z formation - cue the music - baby, we
# were Living La Vida Loca!
#
# So along comes the SES sale to a no name company that no one
# stateside had every heard of and who had no history in this space.
# Most analysts felt that the sale price of $43 million was pretty
# cheap at the time. The public explanations about it didn't match the
# reality of the sale. Alan Meckler is as wise an old owl as there is
# on the net today. This stuff doesn't happen in a vacuum. There was
# another shoe to drop. Befuddled and not a smoking gun in sight.
# That left the only plausible explanation to be the simple face value
# one: that $43 million in the hand is better than the vast unknown in
# the future bush.
#
# We had a sharp consultant working for us a year ago. She was helping
# with a long term business plan and strategy. She did a study of the
# seo/sem/conference sector. Her work, was all top notch. Stuff we
# read about in business books, but we never thought we could actually
# have completed. As she was presenting the competitive Intel on
# Jupiter/Incisive to us, she talked a bit about the sale of ClickZ/SES.
# She made a comment that now seems pretty insightful:
#
# "I would be curious to see Danny Sullivans contract. It is
# the only variable we don't know about."
#
# As they say; that knowledge and $3.75 will still buy you a small
# coffee at Starbucks. The only thing we could do was go back to work
# and continue improving and producing our sites and conferences. We
# always knew there was going to come a day when "the rest of the
# story" would come out. We put our heads down and focused on our work
# - that is all we have ever tried to do. Competition will always be
# there. One of my all time heroes Jonny Carson told Dave Letterman
# when he was in 3rd place of 3 talk shows to: "forget about the other
# guys and what they are doing. Focus on your thing, your audience and
# what you do best". I had a full plate running my own business and
# didn't need to dedicate any more brain power to thinking about it. I
# knew it would work out. This too shall pass.
#
# So, Yesterday when I read the news that Danny was leaving Search
# Engine Watch, I honestly was not all that surprised. 13 months
# later, Alan Mecklers other shoe hits the floor. I finally exhaled.
#
# From there, you lead into all kinds of questions about the state of
# Danny's contract going back more than a year to the time of the sale.
# psst: Apparently, he did not have a noncompete. Hello!? I can only
# imagine he refused to renew at some point or the right phase of the
# moon... Ack, stop that. Here we go again with margin calls and ex
# wives... So lets get off that track and say we did but we didn't explore
# those possibilities. As much guilty pleasure fun as it would be to play
# "what if", I have some sanity to think about you know. We'll leave
# the rest of that as an excersize for the reader and ask good old
# Bones McCoy to bail us out: dam it Jim. I'm a conference
# producer - not a mind reader!
#
# Lets cut to the chase: $43 million for SEW/SES/ClickZ without Danny
# Sullivan locked into a long term contract and no non-compete? (belly
# laugh) Raise a virtual glass to Mr. Alan Meckler. You old dog you -
# nicely done sir! Daddy warbucks is back at the top of my list!
#
# http://weblogs.jupitermedia.com/meckler/archives/014664.html
# http://weblogs.jupitermedia.com/meckler/archives/009626.html
#
# I know for about six months I have been increasingly concerned about
# the industry. There was something soothing and calming about SES San
# Jose, but it seemed so status quo to borderline "been there - done
# that".
#
# I've been thinking the industry was going to see more rapid change
# again this year. Last year was about consolidation and acquisition.
# This year has been about, huh? Not much really. There has been a
# vacum. No defining moment until now. So far, that hole is being
# filled with the Web 2.0 talk. Hey, Web 2.0, is proof that people need to
# talk about something.
#
# Me and a good friend were chatting a few months ago that we knew we
# were probably going to see some shake up at SearchEngineWatch this
# year. You see, Danny Sullivan just turned 40 this year. I don't care
# who you are, the big four oh works on a man in weird ways. I know it
# does. I've been there recently. I have the new convertible sports
# car to prove it. We thought Danny would get a dog or something - not
# leave SEW!
#
# All the possibilities are swirling. It reminds me of something I
# have never talked about much. Six years ago I was faced with a
# difficult choice between the words: SearchEngine and the word
# Webmaster. Those two domains SearchEngineWorld and WebmasterWorld
# were both powerful in my mind and both were/are extremely brandable
# domains. I have been telling myself that I choose WebmasterWorld
# because I wanted to go horizontal and not vertical with the topics.
# I wanted to cover both search and webmaster related site operation
# and management. I still feel they are not divorceable from one another.
# You can't talk search, without immediatly getting to page and then
# server issues. That intention still covers the basics of my
# reasoning at the time and without that desire I never would have
# chosen WebmasterWorld over SearchEngineWorld. I knew that I was
# choosing the one that would ultimately be the horse we would ride. I
# realize that some of that decision process was the gnawing fact that
# I didn't want SearchEngineWorld to compete with SearchEngineWatch. I
# didn't want to muddle the waters of branding. I can distinctly
# remember trying to find another domain because I didn't want the
# branding confusion. When I first registered SeearchEngineWorld, I
# just thought it was a cool name. Even in 99, Danny had done a great
# job at positioning SearchEngineWatch as a good brand. I never
# intended to see SearchEngineWorld wither and get forgotten. That was
# more from the fact I could never find my Chris Sherman, than any
# other factor. Guys like Chris don't grow on trees.
#
# http://blogs.guardian.co.uk/technology/archives/2006/08/31/dont_go_danny_we_love_you.html
#
# I think the change may do us all some good. Other than our reaction
# to it, we dont' have a big choice in the matter. We need a bit of a
# status quo shake up. The conferences and the sites are getting
# fairly mature. Change is ultimately good.
#
# I doubt Danny will have any problem finding work (grin). I can only imagine
# he has a tough task of deciding what he wants to do next. Think
# about it for a minute: Imagine you could do anything you want with
# almost anyone you wanted to in your industry? The world is your
# oyster. How would you decide? Ultimately, I think Danny will follow
# his passions and the things he finds most fulfilling in life.
#
# It is also interesting to to see that Susan Bratton is leaving as
# the Chair of Ad-Tech. I believe we will see Susan pop up somewhere
# very soon. When I first met Susan, it was a huried introduction at a
# conference. I recognized the look and the brusque stick-to-business
# style. That is the same look I see in clips of me from our
# conferences. At the time, I flat out - umm - didn't get her. Then
# Mike 'rock on' Grehan took us all out for dinner in London. We
# had a nice time and talk with Susan. Then I stumbled on to one of
# her shows on WebmasterRadio and she started to make perfect sense.
# Maybe she has found her forte in Radio instead of conferences?
# Either way, her rolodex is a whos-who of the entire marketing
# industry. I doubt she'll have trouble finding work. From the whispers
# of the rumor mill, Susan was never highly compensated at Fad Tech. I
# could see her doing other conferences soon. However, I wonder if her
# contract allows it?
#
# http://blog.dmnews.com/2006/07/24/drew-ianni-succeeds-susan-bratton-as-adtech-chair/
#
# As for Incisive, this looks like it was a simple 1 plus 1 must equal
# 2 business decision. It sounds like Danny was already making top
# dollar on the circut and Incisive thought any more would be
# financial suicide. Although we don't know alot about Incisive on
# this side of the pond, they do alot of conferences and small trade
# shows around the world. They are very well versed in the economics
# of conferences.
#
# As for SES, it is an intersting pickle Incisive is in. The large
# sponsors are locked in contracts for another year. That's Mr.
# Meckler's handy work in action again. Before Alan brought that to Search
# conferences, there were only a few super large conferences that
# could get sponsors to lock in for a year at a time. That means that
# SES is directly funded through SES San Jose next year. Contracts
# are reserved on the conference space and other conference associated
# services. You can't just show up at a Hilton in New York or the San
# Jose convention center without a year+ advance reservation.
#
# Regardless of the over all appearance or make up of those
# conferences - those conferences are locked in for the most part. We
# have also heard (know) that many speakers also have had their
# expenses covered to these shows. A few of them go back many years on
# the circut. Some, simply wouldn't have a career, let alone a
# speaking slot if it weren't for Danny and SES. Many will continue
# to ride the SES horse for as long as it has legs. Then there are the
# attendees. Most of the NYC crowd is corporate Madison Avenue. Same
# goes for San Jose. What that means, is that many of those people
# already have it in next years budget to attend those conferences and
# whether DS's smiley face is up in front or not, there will be a
# sizeable set of attendees present.
#
# Then there is the real value of a company and conference - the
# people. The SES crew is strong. Karen and the gang have done an
# incredible job over the years. I know just how important and
# valuable those people are and I have the highest regard for them.
# Trust me - running, planning, and executing a conference the size
# of SES is no easy task and experienced people are very few and far
# between.
#
# I think Chris Sherman has been contributing a ton of stuff to SES
# over the last few years. He has organized many panels and session
# tracks - exclusively doing many of them.
#
# What I am getting to - tipping toeing around - is that for the short
# term over the next year, SES doesn't flounder too far. Sponsors,
# exhibitors, speakers, and attendees - it's a dead lock. Attendance
# may suffer, and the quality of the sessions may suffer - but the
# beat goes on. Incisive has six to nine months to figure it all
# out. That is if Danny doesn't throw a curve ball their way.
#
# Where Incisive is truely going to miss DS, is on the site.
# SearchEngineWatch *is* Danny Sullivan. And despite all the blog
# stuff, guest writers and moderators - whatever dude -
# searchenginewatch is Dannys off spring. Always has been - and always
# will be...
#
# For our part - fresh opportunies open up. The phone has been ringing
# alot this week. The popular question has been about speaking in
# Vegas and next years conferences. Yes, we still have few slots to
# fill for Vegas.
#
# http://www.pubcon.com/session_proposal.cgi
#
# All-in-all, things will work out ok I think.
#
# Brett



# =========================================================================== #
# #
# BestBBS 4.0 #
# #
# by brett tabke #
# 7/8/2006 #
# #
# =========================================================================== #
#
#
# We have just completed a significant update to the forum software
# (BestBBS) that we run on WebmasterWorld.
#
# The majority of this update was an internal database format
# conversion. It took well over 300 hours of labor to convert the
# system to the new formats. When completed, we have a faster, more
# secure system than we have ever had before.
#
# The change was necessary to fix a long standing database design
# issue. The problem was exposed as the shear scale of WebmasterWorld
# grew to what it is today. I first wrote the database routines for
# this software in 1998. I felt it would handle 10,000 messages. At
# the time, that was an enormous amount of messages for a web based
# bbs/forum system. The system now has over 2 million. The old system
# consisted of two files for every thread. We have converted that to
# one file per thread.
#
# I have often been asked why we did not switch to a true database
# system such as SQL. The issue has always been one of speed. The
# issue has been negated somewhat due to faster machines. However, the
# core of WebmasterWorld is still written in Perl - which is
# notoriously slow at database routines - when compared to languages
# with builtin C bases SQL routines such as PHP. That said, we have
# laid in provisions for major future changes (read: SQL) to all the
# database routines.
#
# Unfortunately in spite of all that effort, there is little to be
# seen new on the user side for the moment. The majority of these
# changes were under-the-hood.
#
# The most visible change noticed by members will be the new keyword
# URL system. We maintain the old files at their existing urls, while
# at the same time introducing new keyword based urls for the future.
# For example:
#
# Old url:
#
# http://www.WebmasterWorld.com/forum30/
#
# new URL:
#
# http://www.WebmasterWorld.com/google/
#
# It is significant to note, that your old bookmarks to specific
# discussions will continue to work. However, the index pages for each
# forum will slowly be changed over to the new keyword URL based
# system.
#
# Another thing you will notice, is the spartan look of pages when
# logged out. By not looking up member info on each message, we remove
# a massive chunk of overhead on the system.
#
# More here:
# http://www.webmasterworld.com/webmasterworld/3000001.htm
#
# -bt


# =========================================================================== #
# #
# The PayPal Wars #
# #
# by brett tabke #
# 6/27/2006 #
# #
# =========================================================================== #
#
# disclosure: I am on the PayPal developers advisory board.
#
# I was just studying the GBuy thread closer than I have looked
# and studied an issue in years. This could be the biggest
# competitive struggle we have seen since Microsoft vs Netscape.
# http://www.webmasterworld.com/forum86/4534.htm
#
# Google vs Microsoft/Yahoo/AskJeeves, is/was for the shear horse race
# and sportsmanship value. I feel like the community has a clear
# interest in the outcome, but as a whole - regardless of the battle -
# we will continue and our outcome/survival is assured.
#
# GBuy vs PayPal/Ebay though...this is a different kind of front here.
# Really for the first time, Google is poised to step into unknown
# water with ZERO experience at a real "all grown up" playing field.
#
# When we look at some of Googles bigger offerings, a trend towards
# geekdom comes out:
#
# Gmail - email systems are so 1996 with off-the-shelf programs
# available. They are 'techie' things.
# Maps - there we are with window dressing on a Microsoft (Terra
# server) product from 97.
# Online Advertising buying/selling are essentially off-the-shelf..oldschool.
#
# What else? When you look down the list of G offerings, there
# is little that Google has done up until now that is not purely
# "net related nerdvana stuff".
# http://www.google.com/intl/en/options/
#
# But payments? This is an odd-man-out - a sunflower in the
# daisy patch. I can't help but remember back to when Red Had IPO'd
# and all the *nix geeks freaked out and wet themselves trying to
# get in on the action or even understand it. Many of those same
# geeks now work for Google. Have they learned anything?
#
#
# Apparently - Googles main competitor for GBuy/GPayments will be
# Ebay/PayPal. How ironic that the last month, I have been using spare
# time to read, "The PayPal Wars" by Eric Jackson.
#
# The most striking thing about the PayPal story is just how deep into the
# financial wars PayPal got. Their attempt at international monetary
# domination was meet with a full frontal assault by everyone from the
# powerful financial communities to the Russian Mafia.
#
# ------------------------------------------------------------ quote:
# PayPal looked like a gold mine for crime rings as well as
# sophisticated independent crooks trafficking in stolen credit cards.
# ...With a CD-Rom full of stolen numbers and a robotic script designed
# to open PayPal accounts, Internet-savvy criminals could easily
# automate the creation of hundreds of thousands of dummy users. Those
# feeder accounts could then use the stolen credit cards to send
# payments through a layer or two of additional fraudulent accounts
# before the criminal initiated an ACH to transfer the balance out of
# the system.
# ----------------------------------------------------------- /quote
#
# To say that PayPal is the most battle hardened dot com on the
# web today is an understatement:
#
# ------------------------------------------------------------ quote:
# When the Russian and Nigerian mafias rung up online charges,
# they ultimately plundered PayPal, not the cardholder. While our
# customer base continued its explosive growth, these brazen criminals
# walked in through our front door and carried on their activities
# largely unmolested. In what we would later refer to as "a
# significant fraud episode," one such fraud ring cost the company
# $5.7 million over a four month period in mid-2000.
# ----------------------------------------------------------- /quote
#
# From there we finally get some long over due answers from the PayPal
# side of things. One doesn't need to dig too deep on the net to find
# some very PO'd people teeing off on PayPal. There were a few, but
# vocal group of people that had their PayPal accounts frozen in the
# midst of the all out fraud assault on PayPal. It was always assumed
# that those accounts were some how caught up in the fraud schemes
# PayPal was fighting. For the first time, we have a few comments from
# PayPal on the story:
#
# ------------------------------------------------------------ quote:
# PayPals success in fighting back fraud also produced false
# positives that inconvenienced honest users.... But as bad as the
# false positives experience for innocent users and resulting negative
# publicity for the company might have been, it was an acceptable
# cost. The fact that spiraling fraud losses contributed to many of
# our competitors, like eMoneyMail, PayMe, and PayPlace ceasing
# operation made this an easy choice. Had PayPal not found a way to
# get fraud under control, it would have destroyed the company.
# ----------------------------------------------------------- /quote
#
# Whatever the outcome - this is going to get interesting. PayPal/Ebay
# are clearly not a bunch of sheep.
#
# If you are looking for some insight into GBuy vs Ebay, I would read
# the PayPal Wars.
#
# Questions still remaining:
# - Has PayPal still got fight in them?
# - Has Ebay Neutered PayPal?
# - What could Ebays possible response be?
#
#
# bt
#
#...The early bird gets the worm, but the second mouse gets the cheese.
#


# =========================================================================== #
# #
# Wavers - Webmasters That Rode The Google Links Wave #
# #
# by brett tabke #
# 5/18/2006 #
# #
# =========================================================================== #

# A few moderators were asking if there was an over-abundance of dissatisfied
# Google users these days?
#
# Riding the Google wave for a whole host of newbies was easy. They cracked
# the Google algo :-) For the first five years of Google, you could
# effectively sum up 80% of the Google algo this way:
#
# =======================================
# Google Algo = Get Links = Good Rankings
# =======================================
#
# That was the SEO expertise of an entire generation of Google Wave Webmasters
# (I call them "Google Wavers" or just "Wavers").
#
# I talked with the LEAD senior optimizer for a top 5 seo firm last summer at
# a conference. We were talking about my Keyword Density Analyzer utility that
# is available here for subscribers. I raised an eyebrow as he did not
# understand concept of keyword density. He simply didn't know what it was and
# instead wanted to talk about links. That is not the exception - that sort of
# ignorance is rampant out there. There are guys and gals reading this right
# now, that are top 50 seo's. There are also people you read every day as
# "experts" who couldn't pass an SEO101 class. I know one seo who has written
# a successful book about SEO, who I am almost positive has never done any SEO
# but her/his blog.
#
# The majority of the SEO knowledge out there today is about links. From
# services that provide linking, to sites that sell links, we have a huge
# investment in "links=seo". So huge, that even long after linkage as SEO is
# dead an buried, there will still be those trumpeting it as the
# end-all-be-all of SEO to make coin off it.
#
# What happens to all those Wavers that think [i]Getting Links = SEO[/i] when
# that majority of the Google algo is devalued in various ways? Wavers built
# their fortunes on "links=seo". When that goes away, the Wavers have zero to
# hold on to.
#
# All they can do - is complain. An entire generation of webmasters are forced
# to go back to SEO101 and learn what they ignored or never had the fortitude
# and passion to learn.
#
# -bt
#
# ...I have a friend who's a billionaire. He invented Cliff notes.
# When I asked him how he got such a great idea, he said,
# "Well first I.....I just....to make a long story short..." - Stephen Wright
#



# =========================================================================== #
# #
# Password Hell! #
# #
# by brett tabke #
# 5/15/2006 #
# #
# =========================================================================== #

# In one of the more amusing articles, I have seen in awhile,
# silicon.com proposes a set of password guidelines.
# While some are very useful - other are dubious - and still others humorus.
#
# http://www.silicon.com/research/specialreports/idmanagement/0,380001
# 1361,39158933,00.htm

# My take:
#
# 1- Passwords must not be written down
#
# Yes, you must remember that autogenerated forced password like:
# "71JU28kIjjL7126" by pure memory power alone. LOL!
#
# Fact: There is little chance of your remembering all the passwords
# you are forced to use.
#
# Tip: Always record your passwords somewhere in some form. We'd
# recommend one of the many password keeper programs that can be
# locked. Or, write them down and put them in a company safe. We'd
# suggest a simple encryption system such as exchanging the first or
# last two characters of every password.
#
#
# 2: Passwords must be set
#
# TIP: If a required secure system allows you to leave a password as a
# default (such as password or admin) - then get a new system because
# the current one is not secure and there is little doubt that
# fundamental security flaws will be found under the skin.
#
# 3: Require as few passwords as possible
#
# FACT: You have no choices in this matter. You gotta use - what you
# gotta use.
#
# 4: Staff must change their passwords regularly
#
# That is a good tip, but fred2006 will change his password to
# fred2005 and flip back and forth. Most studies on this one have
# shown that people simply cycle between a few default flavors.
#
# 5: Make new passwords new
#
# The only way to do that is to use a randomization routine. Which
# brings us full circle back to #1.
#
# 6: Avoid obvious words
#
# Good tip. Dictionary attacks still happen. Which means most systems
# should use a 3strikes and you are out for 15mins programming rule.
#
# 7: Think long - but not too long
#
# Password problems are directly proportional to the length of
# passwords. As password length increases - so do support and system
# help calls.
#
# 8: Automate password changes
#
# Good tip, but be sure to lay in more customer support personnel on
# the days you force password changes. Sally in accounting is sure to
# call with password problems. Be sure to lay in more training for
# your support personnel, because as calls increase, so do social
# engineering related hacker calls.
#
# 9: Educate staff
#
# Deja vu.
#
# 10: Look to the future such as biometrics and two-factor
# authentication
#
# That's great - beam me up Mr. Scott.
#
# Fact: real world biometrics are 5-10 years away. Todays finger print
# scanners have a high failure rate and are not workable in the real
# world.
#
# The closest we have are the retinal eye scanners. They have quite a
# r&d cycle to go before ready. They run as high as a 25% failure
# rates right now, but the future looks very bright for them. They
# might also be combine with health scanners to spot the early onset
# of eye diseases.
#
# Warning - software plug:
# I use SplashID for desktop and for my phone. Works great...
#
# -bt
#
# ... For my birthday I got a humidifier and a dehumidifier. I put
# them in the same room and let them fight it out.


# =========================================================================== #
# #
# Digging Out from Boston #
# #
# by brett tabke #
# 5/4/2006 #
# #
# =========================================================================== #

# Whew, finally have dug out from the onslaught of work that was Pubcon Boston.
#
# What a great conference it was. The blogging session was my favorite I think.
# Setting in with Matt and Robert, and Jeremy was like being allowed to listen
# in on a private conversation between the three of them. Funny thing was that
# Robert barely made it to the conference. The cab driver took him to the wrong
# convention center.
#
# Attendee numbers were on par with last years New Orleans conference. We really
# had to fight for it thought with the proximity to Easter and the other big
# conferences going on at the same time.
#
# Special thanks to Malcolm Gladwell for helping with charity book signing.
# Together we raised nearly $3000 for charity.
#
#
#
# Brett
#
# ... It doesn't matter what temperature a room is, it's always room temperature.

# =========================================================================== #
# #
# Follow The Money #
# #
# by brett tabke #
# 3/28/2006 #
# #
# =========================================================================== #

# <advert>
# ooooooooooooooooooooooooooooooooooooooooooooooooooo
#
# WebmasterWorlds PUBCON BOSTON
#
# Keynote by Malcolm (Tipping Point) Gladwell
#
# Get the PubCon Edge!
#
# http://www.PubCon.com
#
# ooooooooooooooooooooooooooooooooooooooooooooooooooo
# </advert>

#
# When not conferencing or tweaking servers, the great state of
# "newbieness" is my top "blue sky" topics to contemplate.
#
# I can not count the times that people have come up to me at the
# conferences and said something to the effect:
#
# "3 months ago I was working in a factory....now I am making more
# money and working much less thanks to WebmasterWorld." There are
# plenty of newbie stories out there just like that.
#
# Remember in the 80's when everyone was waxing on about "the life
# cycle" of the human? If you understood the life cycle of humans you
# could make a fortune (eg: knowing to invest in fitness equipment
# when the baby boomers hit 30, or in luxury cars when they hit 40, or
# investment schemes when they hit 50, or retirement stuff when they
# hit 60).
#
# I think to really understand the current state of newbieness, you
# have to understand the life cycle of the web user:
#
# The CIA says something to the effect, "that 88% of all humans will
# live 99% of their lives within 25miles of the spot they were born".
# Most of us were born on Yahoo and we have not strayed to far from
# the nest. Most of what we do on the web today is still filtered
# through those early Yahoo colored glasses. Even Google took it's
# starting low impact page que from Yahoo.
#
# Today, many new web users and new webmasters are being "born on
# Google.com" and continue to ride the Google wave into webmastering.
# Understanding the new newbie webmaster phenom is to understand the
# Google wave is to understand the Newbie wave. Circular? Yes it is.
# The bottom line is that it is Adsense - it is the heart and soul of
# the newbie wave. I don't think we have seen anything yet. There is
# an explosion of newbieness going on out there and we are just
# scratching the surface here.
#
# -bt
#
# ...You want to know what is going on? Then follow the money - (Deepthroat 1973).
#


# =========================================================================== #
# #
# What is Your EQ? #
# #
# by brett tabke #
# 3/14/2006 #
# #
# =========================================================================== #

#
#
# Sorry for the delay in updating - I got tied up in this tornado of
# exuberance called PubCon prep work.
#
# Good friend digitalghost tipped me off to a fresh business quiz.
# http://www.webmasterworld.com/profile.cgi?action=view&member=digitalghost
#
# This one is by noted marketing guru Guy Kawasaki:
# http://blog.guykawasaki.com/2006/03/whats_your_eq_e.html
#
# ----------------------------------------------------------------------
# Here's a quiz to determine your "entrepreneurial quotient". My
# intent is to test a person's knowledge of entrepreneurship. However,
# scoring high doesn't mean you're the next Steve Jobs, and scoring
# low doesn't mean you're not. Some answers are debatable, so there
# will be many comments.
# ----------------------------------------------------------------------
#
# ok - haven't taken the test just yet. I got to reading that page
# and several things besides the test freaking leaped off the page and
# need major comment.
#
# a) guy is pretty cool. We thought about him for a keynote at PubCon
# and may have him in the future. I saw him speak a couple years ago
# in Chicago and he is a very righteous speaker.
#
# <satire>
# b) What is that over on the right where the traditional trashcan
# blog roll should be? What is a Tag Zoom cloud? That looks brilliant!
# Where have I been on that one...wow. Take your tags and dump keywords
# on to the page and don't call is page spamming - call it a
# Cloud! (grin)and people wonder why tagging teams are going at it?
# </satire>
#
# c) Another Text-Link-Ads ad on that page. Patrick Gavin of TLA is
# without-a-doubt this last years genius advertiser in the SEO space -
# he is EVERYWHERE with his product. What and excellent job of
# brand building with a sometimes controversial product. Nice Work!
#
# ok - back to the test...
#
# I missed 7.
#
# #4: The foundation of a successful brand is:
#
# I still think should be A (Effective marketing). It was a trick
# question because a great product does NOT equal a great brand. But,
# a bad product can equal a great brand (eg: Ipod was 2 years late to
# the market, major under powered, 50% higher than competitors, and
# broke alot in the early days...)
#
# #5: Ultimately, who positions a product or service establishing how
# customers will come to view it?
#
# The company positions a product from start to finish. What happens
# in the market place by the consumers - is not positioning. Any
# company that believes the product, the message, and the positioning
# is out of their control - is doomed - go home people - it's over.
# You control EVERYTHING. Yes, there comes a point when every child
# leaves the nest, but even then, they can phone home to get money
# wired after that weekend bender at college. The point being, that
# a culture of accountability, resposibilty and total empowerment
# must live in the product from birth until death pulls if off the
# shelf. Nothing less than a employeement force that takes total
# ownership for a products life cycle is acceptable.
#
# #6: If you want your company to be successful, it's most important
# to strive for which objective?
#
# Ok fine, life, liberty and the pursuit of a monopoly.
# Tastes like chicken.
#
# ....meanwhile, back in the real world.
#
# Ya, i will cut Guy some slack on this one. I know where he is coming
# from, however I do not believe that you HAVE to be the sole
# provider of something people really want to be profitable, successful,
# or even retire early. All of that can be accomplished in
# a highly competitive market.
#
# #10: Which part of a business plan is the most important?
#
# Well, he warned us that it was trick question in his intro. lol.
# executive summary. Have you ever seen a business plan that had an
# "executive summary"? (shrug - yes I have) In lieu of that - the
# competitive analysis is most important.
#
# #13: A company that is bootstrapping should avoid which management
# practice?
#
# I suppose this is a little too close to home for me to see the
# answer clearly. I said C (Positioning against the industry leader),
# but b (Trying to recruit a "dream" management team of proven
# executives) makes very good sense. Chimps can manage in the early
# days and you need committed, passionate heavy lifters to lay down
# the schema and carry out the existing plan as growth overwhelms you
# in the early days.
#
# #15: You've just met with a key potential account. It could be a
# large sale and also bolster your company's credibility in the
# industry. However, the account is afraid to do business with a
# "startup." The best way to win them over is to:
#
#
# 15: I totally knew his answer was going to be C: Offer to do a
# pilot implementation at a deep discount.
#
# Unfortunately, I know alot of companies that have held going out of
# business sales because they said C too. Just talk to companies like
# Novell, Stacker, Word Perfect, Lotus, Apple, or Geocities about
# showing people a "demo" at a deep discount. Never heard of some of
# those companies? Ya, they all are about dead or suffered extreme harm
# at some point, because they happened to give people deeply discounted
# demos. There is no better way in todays world to get your product
# ripped off than to give discounted demo. Where would Apple be today
# if Steve Jobs hadn't given a mac demo to Microsoft and went with
# some other software house?
#
#
# - BT

# =========================================================================== #
# #
# The Left and the Right Wing Assault Four Big Horses #
# #
# by brett tabke #
# 2/15/2006 #
# #
# =========================================================================== #

# <advert>
# ooooooooooooooooooooooooooooooooooooooooooooooooooo
#
# WebmasterWorlds PUBCON BOSTON
#
# Keynote by Malcolm (Tipping Point) Gladwell
#
# Get the PubCon Edge!
#
# http://www.PubCon.com
#
# ooooooooooooooooooooooooooooooooooooooooooooooooooo
# </advert>


# If you happened to watch todays hearings that featured Google, Yahoo,
# Cisco, and Microsoft you saw a double sided ambush. These are some folks
# worried about their congressional seat in an election year. These
# are same lot who voted for open and free trade with China. Yes, this is
# the same lot that is now waxing poetic and pointed accusatory fingers at
# the big four for doing what we do - making money in the market place
# that our government and lawmakers created by granting China, most favored
# trade status.
#
# The message underneath is pretty clear : "as long as it political
# prisoners working for 50cents a day to build designer tennis shoes
# for Walmart in Shanghai - it is ok. But once it is "in your face" with
# flat out censorship - it is not ok! There is zero difference between
# that guy building Nikes for Walmart and Google selling the keyword
# "great wall".
#
# Dear Washington;
#
# Get over it, or change your damnable trade laws THAT YOU MADE.
# Leave the internet alone. It is too big and complex for you to
# understand. Let the boys in California do business and quit trying
# to make political capital out of a less than ideal situation. It is
# not up to the Four Horsemen to fight our national battles or more
# precisely - the battles you don't have the nads to fight where you
# should be fighting. If you have a problem with the way China is
# doing business, then take it up in Beijing.
#
# Brett Tabke
##


# =========================================================================== #
# #
# Head Down and Spending Time With The Cube #
# #
# by brett tabke #
# 2/14/2006 #
# #
# =========================================================================== #

# First, happy Valentines day in the US. I have spent the better
# part of the last few weeks - locked in combat with "the cube". Yes,
# it is coming up on conference time (www.pubcon.com) and I am busy
# finishing the session grid. I have always likened it to solving a a
# rubics cube. XX number of Sessions with XX number of speakers with
# XX number of egos ;-) makes for a fun challenge. The good news is -
# the grid is almost done! Best conference ever.
#
# -bt



# =========================================================================== #
# #
# The Richter Scale of Conferencing #
# #
# by brett tabke #
# 2/1/2006 #
# #
# =========================================================================== #

# <advert>
# ooooooooooooooooooooooooooooooooooooooooooooooooooo
#
# WebmasterWorlds PUBCON BOSTON
#
# Keynote by Malcolm (Tipping Point) Gladwell
#
# Get the PubCon Edge!
#
# http://www.PubCon.com
#
# ooooooooooooooooooooooooooooooooooooooooooooooooooo
# </advert>

# post note: Did I just become the first one to ever put a banner ad in a robots.txt?

#
# re:http://www.buzzmachine.com/index.php/2006/01/29/exploding-the-conference-business/
#
# Last week Jeff Jarvis (buzzmachine - one of my 'gotta reads') talked
# about conferences in a blog post. His sentiments have echoed many I
# have heard. Scoble addressed many of Jeffs comments, but even Robert
# was a bit conservative in his estimates of what a conference
# costs these days.
#
# http://scobleizer.wordpress.com/2006/01/29/conferences-vcing-hot-topics-this-morning/
#
# 2000 of My Closest Friends for 3 Days:
#
# Lets say you are going to plan a conference for 2000 people for 3
# days. To hold a conference that size, you are going to be pretty
# limited in what venues you can use. Those that will work - will cost
# you. If you are in a hotel, your rates will go down, but in a
# convention center they can increase significantly. At a minimum, you
# will pay about $1 a square foot. Or, a conference of 2k for 3 days,
# will run $50,000 to $80,000 for the space.
#
# Next, a you will need a contractor to handle the conference setup,
# and to work with the venue. Most cities and convention centers
# require union labor to handle things like booths. That will run you
# another $20k-$50k.
#
# You will need some AV. You can do it for about $10k on the cheap, or
# $20-$30k if you wanna be real.
#
# Then you'll need security as mandated by the venue. About $5-7k.
#
# Then some labor to operate the thing. Another $10-15k.
#
# Conference books. About $10-15 per attendee ($20-$25k).
#
# Shipping all this stuff - another $5-20k (it gets out of control in
# a hurry).
#
# Figuring out how to get all this to sync - atleast $10k in
# consultants (very conservative).
#
# Computers to manage your registration on site and for presentations
# $5k.
#
# On venue site misc expenses (trust me) $10-20k.
#
# Hey, you want internet at your conference? That will run $500 to
# $1000 PER connection! (or $8k easy)
#
# You want a big shot keynote speaker? Figure $40-$150k speaking fees
# (plus expenses).
#
# Showing your panel speakers some love : $2k - xx,000
#
# Then your online site $7-10k a year.
#
# You gotta have some of them there perty signs to tell everyone
# what's where when - $2k on the cheap - $5k to get real.
#
# Then your guys to build and run the conference (1 person per 500
# attendees min) at probably 70k-90k a year.
#
# You need to get your people there right? Lets say $20k in planes,
# trains, taxis, and hotels.
#
# Who is that over there? Billy Joe Jim Bob Name tags: $3 each. ($5k)
#
# Taxes, insurance, offices, phones, internet - oh no!. $xx,xxx -
# $xxx,xxx per year.
#
# Decorators? Don't even go there! $20k just for designs.
#
# Meeting planners? $10k just for the coffe talk. --$50k for them to
# actually do some work.
#
# Still with us? Ok, here is where we shake out the men from the boys:
#
# Getting a hotel for the conference? Lets say you reserve a room
# block of 750 rooms for 3 nights : 2250 times at say $159 a night.
# That's a cool $380,000 you just guaranteed bucko. You still have the
# stomach for this? What's more? you are probably going to have to
# make that commitment 1 full year out before the conference! Hello!?
#
# Oh wait - we were doing for 2k people right? And they will be there for
# the 3 nights of the conference plus the travel day. So that is 8000
# room nights and you will probably spend $180-$225 for rooms. So a
# real figure closeer to $1.25- to $1.8 million. Yes, that is MILLION!
#
# Next you'll need to feed some people. Most conference centers will
# run about $50 per person per day for a continental breakfast and box
# lunch package. Or for a three day conference, $150 per attendee. Any
# where from $300k to $500k for food total. If you get shmancy fancy,
# you could easily top $150 per person per day.
#
# Hey, so you are out of pocket for half a million to one million on this
# "little conference" and committed to several hundred more - you
# best be doing some marketing. add $50-$150k.
#
# Happy Fun Thoughts:
#
# As scary as that may be - lets think some happy thoughts and talk
# revenue baby:
#
# In order to build a conference of 2k, you will end up comp'ing about
# 25%. Those will include staff, speakers, friends, friends of
# speakers, and VIP's. Every conference I have been around has a comp
# rate of 25% or higher.
#
# Then you will have deeply discounted rates for exhibitors and
# sponsors. If you have 50 exhibitors, you can figure on giving each
# 4-6 passes to go along with their package.
#
# Next, you will more-than-likely have a discount rate for exhibit
# hall only attendees. That will run another 10-20% depending on your
# conference style and attraction. Of that, you will probably have a
# discounted coupon rate for your exhibitors. I know of one large
# conference that regularly gives exhibitors a 50% discount rate that
# they can pass on to their clients.
#
# Then you will probably have a deeply discounted early sign up rate.
#
# Lastly, many conferences will have single "day rates" that cut
# prices in half for each day.
#
# So here is the big secret in conferences today - at a minimum, a
# modern conference of 2k will have less than 50% that actually pay to
# get in the door, and of those - only 30-35% will pay full price. For
# example, I know of one big new york conference that had 8k people
# attend last year. Of those, only 555 were paid in full, 1500 were
# single day passes and the rest were exhibit hall only and comps.
#
# Exhibits? Ya, those will require special handling. Rarely do
# conferences sell packages at full price. Even the largest
# conferences out there today often discount their sponsorships 50%
# for multi conference signups.
#
# So, as Scoble hints, it is exponentially harder to do a conference
# of 2k than it is to do a conference of 4-5 hundred. It is like that
# Richter scale thing where things start multiplying out of control.
#
# Our Orlando conference of 500 in Feb of 2004 was 7000% more
# profitable than our Las Vegas conference of 1500 last fall.
#
# I think 2006 will be a big year of shake out in the conference
# business. We will see many smaller conferences fade and the bigger
# ones solidify their standing. The Fad-Techs and the CES's will grow
# in dominance. What will separate the big from the small will be
# marketing.
#
# Does Alan Meckler selling Ses makes a whole lot more sense now?
# Alan will not be missed (atleast by those of us that run
# conferences...lol)
#
# Jeff did have some very interesting comments about a conference being
# a conversation, we will tackle those another day.
#
# BT
#

# =========================================================================== #
# #
# Cache This - Or, Setting Up Your Own Vertical Search Engine #
# #
# by brett tabke #
# 1/27/2006 #
# #
# =========================================================================== #
#
# http://www.webmasterworld.com/forum30/32936.htm
#
# http://www.eff.org/news/archives/2006_01.php#004345
#
# I've said for years that in any court case, "the one with the most
# money will 99% of the time win. Yesterdays ruling that Googles
# republished copy of pages (they call it a "cache") proves that
# point.
#
# What Does It Mean:
#
# It means we can now all start "caching" search engines and
# republishing works. Simply download and install a search engine
# just as ASPSeek and fed it some URLs. And as added bonus points
# The search engines will glady cache your cached pages.
#
# http://www.aspseek.org
#
# - bt
#


# =========================================================================== #
# #
# Another Search Magazine #
# #
# by brett tabke #
# 1/26/2006 #
# #
# =========================================================================== #
#
# The third attempt at a search marketing magazine:
#
# http://www.searchmarketingstandard.com/
#
# Many will probably remember the first big magazine BallyHoo by Troy
# Perkins.
#
# -bt

# =========================================================================== #
# #
# Yahoo, Oh Sorry - She Said it, but We Didn't Mean It! #
# #
# by brett tabke #
# 1/25/2006 #
# #
# =========================================================================== #
#
# http://www.webmasterworld.com/forum35/3819.htm#
#
# Yahoo CFO Susan Dreker said earlier this week:
#
# http://seattlepi.nwsource.com/business/256748_yahoo24.html
# -----------------------------------------------------------------------
# "We don't think it's reasonable to assume we're going to gain a lot
# of share from Google," Chief Financial Officer Susan Decker said in
# an interview. "It's not our goal to be No. 1 in Internet search. We
# would be very happy to maintain our market share."
# -----------------------------------------------------------------------
#
# So when my email and stickymail started going off that Yahoo had
# responded, I was excited. At least until I found out they had
# responded on their blog.
#
# They have the phone number of every major tech reporter in the
# country in their Rolodex. So why not call up Bloomberg for a second
# interview and respond - or how about the AP - or the New York Times,
# - one of a dozen other top news outlets in the country? To publish
# a "nondenial denial" on the corporate blog is to be dismissed from
# top to bottom. It pollutes everything they have to say via that
# avenue from now until eternity
#
# This is a perfect example of why the corporate search blogs
# should be ignored. It is why companies should not act this way.
# It is also why we do not link to corporate search blogs.
#
# If they have something to say - then issue a press release, or
# grab the nearest real reporter on any street corner and talk.
#
# - bt
#


# =========================================================================== #
#
ti89
 
Posts: 44
Joined: 28 Oct 2006 07:37


Return to SEO