XSSFilter could not parse (X)HTML:
<p>From dm.lua at math2.org Sat Aug 23 14:38:33 2008
From: dm.lua at math2.org (David Manura)
Date: Sat Aug 23 14:50:39 2008
Subject: [Sputnik-list] traceback errors, config pages, searching
Message-ID: <a href="mailto:bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com">bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com</a></p>
<p>A few comments...</p>
<p>(1)</p>
<p>When Sputnik raises an unexpected exception, a stack traceback is
displayed on the web page:</p>
<p><snip>
There was an error in the specified application. The full error message follows:</p>
<p>...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:183: cannot
obtain information from file `redirect:/cgi/sputnik.cgi'
stack traceback:</p>
<pre><code> [C]: in function 'assert'
...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:183: in
</code></pre>
<p>function <...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:182></p>
<pre><code> (tail call): ?
...utnik/kepler-1.1/rocks//wsapi/1.0rc1-1/bin/wsapi.cgi:16: in
</code></pre>
<p>function <...utnik/kepler-1.1/rocks//wsapi/1.0rc1-1/bin/wsapi.cgi:14></p>
<pre><code> (tail call): ?
[C]: in function 'xpcall'
...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:135: in
</code></pre>
<p>function 'run_app'</p>
<pre><code> ...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:159: in
</code></pre>
<p>function 'run'</p>
<pre><code> ...k/kepler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/cgi.lua:16: in
</code></pre>
<p>function 'run'</p>
<pre><code> ...utnik/kepler-1.1/rocks//wsapi/1.0rc1-1/bin/wsapi.cgi:26: in main chunk
[C]: ?
</code></pre>
<p></snip></p>
<p>It could be argued that the end user of the web site shouldn't see a
stack traceback. First, there may be security implications in
allowing the end user to know how the web site is implemented and
installed. Second, the stack traceback is more useful rather to the
administrator of a web site, so perhaps it should be recorded instead
to a log file on the server, and end user should only see a ticket
number that the administrator can cross reference against the log
file. I did some searching on this concern just now:</p>
<p>[1] http://www.jankoatwarpspeed.com/post/2008/06/02/Exception-handling-best-practices-in-ASPNET-web-applications.aspx
[2] http://www.securitypark.co.uk/article.asp?articleid=26905
[3] http://www.infosecwriters.com/text<em>resources/pdf/Top</em>10<em>Configuration</em>Security<em>Vulnerabilities</em>Part<em>One</em>BSullivan.pdf</p>
<p>The stack traceback in Sputnik is triggered by error_html in
rocks/wsapi/1.0-2/lua/wsapi/common.lua, so this might instead be a
WSAPI/Kepler concern.</p>
<p>(2)</p>
<p>After installing Sputnik, I had difficulty finding a complete list of
all the configuration pages. Only some were on the start page. I
later discovered they were listed on the "sputnik" page--e.g.
http://sputnik.freewisdom.org/en/sputnik . (BTW, the "_navigation"
link on this page is broken.) I think the "sputnik" configuration
page should be linked from the start page on the initial installation.</p>
<p>(3)</p>
<p>More generally, is there a way to obtain a complete list of all pages
that exist (without indexing them on Google)? Perhaps I'm setting up
a new wiki and want to remove unnecessary pages. On lua-users wiki, I
just enter an empty search in http://lua-users.org/wiki/FindPage .</p>
<p>(4)</p>
<p>I'm quite in favor of adding a built-in full-text search engine that
works out-of-the box, at least as a fallback, even if that may be
inferior in some ways to Google. A discussion about this was here:</p>
<p> http://lua-users.org/lists/lua-l/2008-02/msg00950.html</p>
<p>A potentially common use case is to use Sputnik internally on a small
wiki by an individual or small group. In that case, simple linear
search through the pages (much like grep) would be sufficient and
trivial to implement. More generally, you'd want to maintain an
inverted index, possibly using an existing production-grade search
engine (e.g. http://swish-e.org and others) or Google, but if you want
something trivial to implement now, here's the code used by the usemod
wiki ( http://www.usemod.com/cgi-bin/wiki.pl ), which is the wiki upon
which lua-users.org is based:</p>
<p>sub SearchTitleAndBody {
my ($string) = @_;
my ($name, $freeName, @found);</p>
<p> foreach $name (&AllPagesList()) {</p>
<pre><code>&OpenPage($name);
&OpenDefaultText();
if (($Text{'text'} =~ /$string/i) || ($name =~ /$string/i)) {
push(@found, $name);
} elsif ($FreeLinks) {
if ($name =~ m/_/) {
$freeName = $name;
$freeName =~ s/_/ /g;
if ($freeName =~ /$string/i) {
push(@found, $name);
}
} elsif ($string =~ m/ /) {
$freeName = $string;
$freeName =~ s/ /_/g;
if ($Text{'text'} =~ /$freeName/i) {
push(@found, $name);
}
}
}
</code></pre>
<p> }
return @found;
}</p>
<p>Boolean AND/NOT logic and phrase searching would be a simple extension
to that (e.g. ' "hello world" -goodbye '). You do not need word
tokenization (since there is no inverted index of words) nor stemming,
synonyms, etc., which would complicate the otherwise simple logic.</p>
<p>(5)</p>
<p>When previewing edits to template/config pages, it would be useful for
Sputnik to apply the templates being edited in the preview. This is
especially true since edits to these pages can break the wiki, so it
would be desirable to preview them first.</p>
<p>From jnwhiteh at gmail.com Sat Aug 23 14:54:31 2008
From: jnwhiteh at gmail.com (Jim Whitehead II)
Date: Sat Aug 23 15:06:34 2008
Subject: [Sputnik-list] traceback errors, config pages, searching
In-Reply-To: <a href="mailto:bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com">bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com</a>
References: <a href="mailto:bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com">bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com</a>
Message-ID: <a href="mailto:5fe281d40808230954s7a5979c6sbcdb6ee12ff2f213@mail.gmail.com">5fe281d40808230954s7a5979c6sbcdb6ee12ff2f213@mail.gmail.com</a></p>
<p>On Sat, Aug 23, 2008 at 5:38 PM, David Manura <a href="mailto:dm.lua@math2.org">dm.lua@math2.org</a> wrote:</p>
<blockquote>
<p>A few comments...</p>
<p>(1)</p>
<p>When Sputnik raises an unexpected exception, a stack traceback is
displayed on the web page:</p>
<p><snip>
There was an error in the specified application. The full error message follows:</p>
<p>...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:183: cannot
obtain information from file `redirect:/cgi/sputnik.cgi'
stack traceback:</p>
<pre><code> [C]: in function 'assert'
...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:183: in
</code></pre>
<p>function <...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:182></p>
<pre><code> (tail call): ?
...utnik/kepler-1.1/rocks//wsapi/1.0rc1-1/bin/wsapi.cgi:16: in
</code></pre>
<p>function <...utnik/kepler-1.1/rocks//wsapi/1.0rc1-1/bin/wsapi.cgi:14></p>
<pre><code> (tail call): ?
[C]: in function 'xpcall'
...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:135: in
</code></pre>
<p>function 'run_app'</p>
<pre><code> ...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:159: in
</code></pre>
<p>function 'run'</p>
<pre><code> ...k/kepler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/cgi.lua:16: in
</code></pre>
<p>function 'run'</p>
<pre><code> ...utnik/kepler-1.1/rocks//wsapi/1.0rc1-1/bin/wsapi.cgi:26: in main chunk
[C]: ?
</code></pre>
<p></snip></p>
<p>It could be argued that the end user of the web site shouldn't see a
stack traceback. First, there may be security implications in
allowing the end user to know how the web site is implemented and
installed. Second, the stack traceback is more useful rather to the
administrator of a web site, so perhaps it should be recorded instead
to a log file on the server, and end user should only see a ticket
number that the administrator can cross reference against the log
file. I did some searching on this concern just now:</p>
<p>[1] http://www.jankoatwarpspeed.com/post/2008/06/02/Exception-handling-best-practices-in-ASPNET-web-applications.aspx
[2] http://www.securitypark.co.uk/article.asp?articleid=26905
[3] http://www.infosecwriters.com/text<em>resources/pdf/Top</em>10<em>Configuration</em>Security<em>Vulnerabilities</em>Part<em>One</em>BSullivan.pdf</p>
<p>The stack traceback in Sputnik is triggered by error_html in
rocks/wsapi/1.0-2/lua/wsapi/common.lua, so this might instead be a
WSAPI/Kepler concern.</p>
</blockquote>
<p>I agree 100% and my thoughts were to write the following:</p>
<ol>
<li><p>A configuration page in the core that is displayed whenever a
sputnik-layer error occurs. This is somewhat difficult to manage
considering that the sputnik error may be preventing nodes from being
displayed in the first place. It could be a static page, so I'm open
to ideas on this one.</p></li>
<li><p>A module that sends all errors that occur in the system to some
endpoint so they can be viewed after the fact. There are again issues
with this (how to ensure that the filesystem/db isn't consumed with
these errors, etc).</p></li>
</ol>
<blockquote>
<p>After installing Sputnik, I had difficulty finding a complete list of
all the configuration pages. Only some were on the start page. I
later discovered they were listed on the "sputnik" page--e.g.
http://sputnik.freewisdom.org/en/sputnik . (BTW, the "_navigation"
link on this page is broken.) I think the "sputnik" configuration
page should be linked from the start page on the initial installation.</p>
</blockquote>
<p>All configuration pages, or at least the major sputnik configuration
page should be linked from the main page post-installation so this
just needs to be updated in the node template.</p>
<blockquote>
<p>More generally, is there a way to obtain a complete list of all pages
that exist (without indexing them on Google)? Perhaps I'm setting up
a new wiki and want to remove unnecessary pages. On lua-users wiki, I
just enter an empty search in http://lua-users.org/wiki/FindPage .</p>
</blockquote>
<p>Well there are a few issues with this. Some pages only exist at the
point they are queried and have no transient state, while the rest can
be viewed directly on the file system or whatever backend the
repository is using. I'm not sure if we have a way using LuaRocks to
figure out what modules are possible provided, but that would be the
primary issue there.l</p>
<blockquote>
<p>I'm quite in favor of adding a built-in full-text search engine that
works out-of-the box, at least as a fallback, even if that may be
inferior in some ways to Google. A discussion about this was here:</p>
<p> http://lua-users.org/lists/lua-l/2008-02/msg00950.html</p>
<p>A potentially common use case is to use Sputnik internally on a small
wiki by an individual or small group. In that case, simple linear
search through the pages (much like grep) would be sufficient and
trivial to implement. More generally, you'd want to maintain an
inverted index, possibly using an existing production-grade search
engine (e.g. http://swish-e.org and others) or Google, but if you want
something trivial to implement now, here's the code used by the usemod
wiki ( http://www.usemod.com/cgi-bin/wiki.pl ), which is the wiki upon
which lua-users.org is based:</p>
<p>sub SearchTitleAndBody {
my ($string) = @_;
my ($name, $freeName, @found);</p>
<p> foreach $name (&AllPagesList()) {
&OpenPage($name);
&OpenDefaultText();
if (($Text{'text'} =~ /$string/i) || ($name =~ /$string/i)) {</p>
<pre><code> push(@found, $name);
</code></pre>
<p> } elsif ($FreeLinks) {</p>
<pre><code> if ($name =~ m/_/) {
$freeName = $name;
$freeName =~ s/_/ /g;
if ($freeName =~ /$string/i) {
push(@found, $name);
}
} elsif ($string =~ m/ /) {
$freeName = $string;
$freeName =~ s/ /_/g;
if ($Text{'text'} =~ /$freeName/i) {
push(@found, $name);
}
}
</code></pre>
<p> }
}
return @found;
}</p>
<p>Boolean AND/NOT logic and phrase searching would be a simple extension
to that (e.g. ' "hello world" -goodbye '). You do not need word
tokenization (since there is no inverted index of words) nor stemming,
synonyms, etc., which would complicate the otherwise simple logic.</p>
</blockquote>
<p>I'm sure I speak for both Yuri and myself when I say we would welcome
a contribute module that provides out of the box search for Sputnik.</p>
<blockquote>
<p>When previewing edits to template/config pages, it would be useful for
Sputnik to apply the templates being edited in the preview. This is
especially true since edits to these pages can break the wiki, so it
would be desirable to preview them first.</p>
</blockquote>
<p>I'm not sure how feasible this is with the prototyped/inheritance
system that Sputnik operates under, but I agree this would be useful.
I can definitely see the use case for this.</p>
<ul>
<li>Jim</li>
</ul>
<p>From petite.abeille at gmail.com Sat Aug 23 16:18:48 2008
From: petite.abeille at gmail.com (Petite Abeille)
Date: Sat Aug 23 16:30:54 2008
Subject: [Sputnik-list] traceback errors, config pages, searching
In-Reply-To: <a href="mailto:5fe281d40808230954s7a5979c6sbcdb6ee12ff2f213@mail.gmail.com">5fe281d40808230954s7a5979c6sbcdb6ee12ff2f213@mail.gmail.com</a>
References: <a href="mailto:bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com">bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com</a></p>
<pre><code><5fe281d40808230954s7a5979c6sbcdb6ee12ff2f213@mail.gmail.com>
</code></pre>
<p>Message-ID: <a href="mailto:1E382948-37D2-444D-85D9-C0B5D99C400D@gmail.com">1E382948-37D2-444D-85D9-C0B5D99C400D@gmail.com</a></p>
<p>On Aug 23, 2008, at 6:54 PM, Jim Whitehead II wrote:</p>
<blockquote>
<p>I'm sure I speak for both Yuri and myself when I say we would welcome
a contribute module that provides out of the box search for Sputnik.</p>
</blockquote>
<p>Perhaps SQLite's FTS module would be of interest:</p>
<p>http://www.sqlite.org/cvstrac/wiki?p=FtsTwo
http://www.sqlite.org/cvstrac/wiki?p=FullTextIndex</p>
<p>FWIW, Nanoki provides its own full-text search by implementing an <br/>
inverted index in SQLite:</p>
<p>http://dev.alt.textdrive.com/browser/HTTP/Finder.ddl
http://dev.alt.textdrive.com/browser/HTTP/Finder.dml
http://dev.alt.textdrive.com/browser/HTTP/Finder.lua</p>
<p>Cheers,</p>
<p>--
PA.
http://alt.textdrive.com/nanoki/</p>
<p>From yuri at sims.berkeley.edu Sat Aug 23 16:51:02 2008
From: yuri at sims.berkeley.edu (Yuri Takhteyev)
Date: Sat Aug 23 17:03:05 2008
Subject: [Sputnik-list] traceback errors, config pages, searching
In-Reply-To: <a href="mailto:bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com">bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com</a>
References: <a href="mailto:bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com">bc4ed2190808230938x1652c455yc757bf60059c6349@mail.gmail.com</a>
Message-ID: <a href="mailto:fa4efbc00808231151x41ee96a8j261c93d01de5a77e@mail.gmail.com">fa4efbc00808231151x41ee96a8j261c93d01de5a77e@mail.gmail.com</a></p>
<p>This may repeat Jim's answers a bit.</p>
<blockquote>
<p>When Sputnik raises an unexpected exception, a stack traceback is
displayed on the web page:</p>
</blockquote>
<p>That's simply a bug. We used to have a nicer page and then something
changed in WSAPI and we haven't updated. That said, we should also
focus on avoiding any error messages. :) So...</p>
<blockquote>
<p>...epler-1.1/rocks//wsapi/1.0rc1-1/lua/wsapi/common.lua:183: cannot
obtain information from file `redirect:/cgi/sputnik.cgi'</p>
</blockquote>
<p>...this specific one (which David reported to me separately) has been
forwarded to Fabio and fixed.</p>
<blockquote>
<p>It could be argued that the end user of the web site shouldn't see a
stack traceback. First, there may be security implications in
allowing the end user to know how the web site is implemented and
installed. Second, the stack traceback is more useful rather to the
administrator of a web site, so perhaps it should be recorded instead
to a log file on the server, and end user should only see a ticket
number that the administrator can cross reference against the log
file.</p>
</blockquote>
<p>Agreed. Having the traceback displayed in the browser speeds up the
development tremendously, but perhaps the best thing to do is add a
config variable ("DISPLAY_TRACEBACK") that turns this on, keeping it
off by default.</p>
<p>Jim: for the logging module you suggest, wouldn't lualogging work?</p>
<blockquote>
<p>The stack traceback in Sputnik is triggered by error_html in
rocks/wsapi/1.0-2/lua/wsapi/common.lua, so this might instead be a
WSAPI/Kepler concern.</p>
</blockquote>
<p>Again, this specific stack traceback was a WSAPI issue that has been
fixed. However, the question of whether WSAPI should be displaying
stack traces may be worth bringing up on the Kepler list. Though,
perhaps this is an application-level issue.</p>
<blockquote>
<p>After installing Sputnik, I had difficulty finding a complete list of
all the configuration pages.</p>
</blockquote>
<p>The links on the sputnik wiki didn't help?</p>
<p>My assumption was that people start with the "Installation" page,
which links to "Basic Configuration" (which is also the next item in
the nav bar. "Basic Configuration" links to configuration (the next
tab), which tells you all of your configuration options.</p>
<p>Should something like this sentence go into the default homepage? :)</p>
<blockquote>
<p>http://sputnik.freewisdom.org/en/sputnik . (BTW, the "_navigation"
link on this page is broken.)</p>
</blockquote>
<p>Oops. The node name changed, the page did not. This is part of the
reason why I've been thinking of just moving all the documentation to
the wiki, so it's all in one place.</p>
<blockquote>
<p>More generally, is there a way to obtain a complete list of all pages
that exist (without indexing them on Google)?</p>
</blockquote>
<p>Yes: http://sputnik.freewisdom.org/en/sitemap
And even in the "Sitemap" XML format:
http://sputnik.freewisdom.org/en/sitemap.xml</p>
<p>(you can give the latter URL to google via "Webmaster Tools" so that
Google will know when new pages are added.)</p>
<p>The only limitation (or feature, depending on how you look at it) is
that this only displays pages that were edited at some point, skipping
the default pages. (That is, your "sputnik" page won't be in this
list, unless you edit it.) This could be changed.</p>
<blockquote>
<p>Perhaps I'm setting up
a new wiki and want to remove unnecessary pages. On lua-users wiki, I
just enter an empty search in http://lua-users.org/wiki/FindPage .</p>
</blockquote>
<p>Makes sense. Though, I've never had to do it this way: Sputnik's
default is to store the data in a very transparent way, each node
being a directory inside wiki-data. So, when I did cleanup of that
sort in the past, I've just cded into wiki-data, did an "ls" and then
a "rm -rf" on the nodes I wanted to delete.</p>
<p>Though, for full transparency of data you would want the Git plugin.
With that, each node is a lua file, revisions are git revisions to
that file, and subdirectories are subdirectories. (That is,
"Tickets/000001" would map be stored in wiki-data/Tickets/000001.lua,
and you could see the revision history by just running "git log
Tickets/000001.lua")</p>
<blockquote>
<p>I'm quite in favor of adding a built-in full-text search engine that
works out-of-the box, at least as a fallback, even if that may be
inferior in some ways to Google. A discussion about this was here:</p>
<p> http://lua-users.org/lists/lua-l/2008-02/msg00950.html</p>
</blockquote>
<p>As Jim said, we are all in favor of this, for reasons that you
mentioned and more! If there was a good search system that was easy
to install and had decent Lua bindings, I swear I would write a plugin
for it the next day. (I've discussed this issue a bit with Jim and
Carregal, basically thinking of it in terms of an API that would
subscribe to modifications and then be able to answer queries.)</p>
<blockquote>
<p>something trivial to implement now, here's the code used by the usemod
wiki ( http://www.usemod.com/cgi-bin/wiki.pl ), which is the wiki upon
which lua-users.org is based:</p>
</blockquote>
<p>I've tried this before and it was very slow. However, I have since
have made a simple fix to Saci (commit beda1d7) which improved the
performance dramatically, making this a viable, though still somewhat
slow approach.</p>
<p>Additionally, I've been working on a Sputnik based photoblog
application that is supposed to allow you to browse blog posts and
albums by tag, which is to say that a version of "search" is meant to
be used heavily. I ended up adding an experimental "application
cache" option, which gives Sputnik or Sputnik-based applications an
option to cache stuff until there is a change to the main storage
system. So, my application would then cache the list of items for
each tag from the moment the tag is first queried and until some new
photos or posts are added. However, this feature is not supported by
all versium implementations at this point. (In fact, by none that are
checked in!)</p>
<blockquote>
<p>sub SearchTitleAndBody {
my ($string) = @_;
my ($name, $freeName, @found);</p>
<p> foreach $name (&AllPagesList()) {
&OpenPage($name);
&OpenDefaultText();
if (($Text{'text'} =~ /$string/i) || ($name =~ /$string/i)) {</p>
<pre><code> push(@found, $name);
</code></pre>
<p> } elsif ($FreeLinks) {</p>
<pre><code> if ($name =~ m/_/) {
$freeName = $name;
$freeName =~ s/_/ /g;
if ($freeName =~ /$string/i) {
push(@found, $name);
}
</code></pre>
</blockquote>
<p>And it would be prettier in Lua. :)</p>
<p>In fact, I will try to make an alternative to "sputnik-search" that does that.</p>
<blockquote>
<p>Boolean AND/NOT logic and phrase searching would be a simple extension
to that (e.g. ' "hello world" -goodbye '). You do not need word
tokenization (since there is no inverted index of words) nor stemming,
synonyms, etc., which would complicate the otherwise simple logic.</p>
</blockquote>
<p>Part of me wants to go this route of adding first one little feature
then another, and eventually implementing a great search engine in
Lua. Another part of me wants to finish my dissertation. :)</p>
<blockquote>
<p>When previewing edits to template/config pages, it would be useful for
Sputnik to apply the templates being edited in the preview. This is
especially true since edits to these pages can break the wiki, so it
would be desirable to preview them first.</p>
</blockquote>
<p>It does work this way for CGI, but not currently with Xavante.
(Because in case of Xavante, the same Sputnik instance is re-used as
for the previous call.) I'll look into this, though.</p>
<p>Thanks for all the comments!</p>
<ul>
<li>yuri</li>
</ul>
<p>--
http://sputnik.freewisdom.org/</p>
