Author Topic: Brain in SQLite?  (Read 29577 times)

archerja

  • Newbie
  • *
  • Posts: 21
    • View Profile
Brain in SQLite?
« on: January 13, 2005, 10:20:51 am »
Robert Medeksza,

I see that the "job posting" sticky is gone.  Did you hire someone for the job?  The reason I ask, is that I am experimenting with Hal using an SQLite database file to retreive information.  I have just simple tables in my database, and Hal seems to be able to handle it just fine.

If people are working on migrating all of Hal's brain into a database, I won't duplicate their efforts.  I will just use my SQLite file for my own "knowledge" database.  I was just using SQLite because it is small (one file for whole database) and easy to connect and script.

thank you.

virtually,
Joseph


Everyone,

P.S. As a side note, I would like to thank everyone who is helping other people within the forums. This is a very nice, comfortable forum to be in.  User support, from the community, is very special.

P.S.S If anyone is interested, I could expand on my use of SQLite.

virtually,
archerja

vonsmith

  • Hero Member
  • *****
  • Posts: 602
    • View Profile
Brain in SQLite?
« Reply #1 on: January 13, 2005, 10:30:52 am »
archerja,
Well this *is* the programming forum. Your comments and insights on any A.I. programming are always appreciated. Please enlighten us on your adventures with Hal and SQLite.

I looked at SQLite a little after your first posting. Interesting stuff. Tell us about the implementation in Hal and maybe share some examples.

Thanks,


=vonsmith=
 

archerja

  • Newbie
  • *
  • Posts: 21
    • View Profile
Brain in SQLite?
« Reply #2 on: January 13, 2005, 02:50:42 pm »
vonsmith,

Well, let see...
I am using SQLite version 3.0.8 (http://www.sqlite.org) with a wrapper to allow for use within vbscript and javascript (http://www.republika.pl/roed/litex). The site has examples in javascript. This allows you to open an SQLite database file, do whatever you want (begin, commit, create table, insert, select, delete, etc.) then close it. VERY streamlined. It only takes a little code.

Anyway, to test it, I put together a "brain.db3" file, created a music table, and inserted listings of the top40 from around 1955 to 1985.  Then created a section in XTF1.2 to use it. Here is a very simple exchange:

me: Who sang Hound Dog?
hal: Elvis presley sang hound dog. Fabian sang hound dog man.

I've got it doing a simple "like", so that it will find any occurance of the word (or string). I've also put a few language files in a table, so that Hal can return the word in another language, like so:

me: Translate wife to Spanish.
hal: Wife in Spanish is la esposa. Wife in Spanish is marida[Noun].
me: Translate wife to Italian.
hal: Wife in Italian is moglie[Noun].

As you can tell from the Spanish result, it will give me all listings for the word. Right now, I only have Spanish, French, German, Italian, and some Portuguese. I wrote a small vbscript to run outside Hal to import delimited text files into my brain.db3.

I was thinking about moving all of Hal's brain over to SQLite. That way, it would be more portable (take her to work with you, then take her home at night). Also, since SQLite is ported to MANY OSes, the input and output parser could be written in the programming language of choice for that OS. (And the Hal community would grow to include Linux, MAC, etc.)

Since I have kids, I was also thinking about how to control different versions of Hal (PG, PG-13, R, XXX, etc.). If I placed Hal's brain into a database, I could have a field for...say "personality key" or pkey for short, then Hal would only return data with the right pkey.  Change your pkey for work or in public, then change the pkey to make her raunchy as hell when your alone.

I had left vbscript a few years back, so I had to dig out some of my old scripts to familiarize myself with the language again. If I moved all of Hal's brain to SQLite, I think I could use (and everyone else) the hal5_XTF_v1.2.uhp for the parser.

Sorry I got so long-winded.  I'll shut up, and listen to some of the responses.

virtually,
Joseph
virtually,
archerja

archerja

  • Newbie
  • *
  • Posts: 21
    • View Profile
Brain in SQLite?
« Reply #3 on: January 17, 2005, 09:15:40 am »
vonsmith,

I would like to try an experiment and TRY to move your XTF over to SQLite.  I will wait until XTF 1.4 comes out. Can I have your thoughts on this?  Good, bad, waste of time, downside, upside, etc., etc. Thanks.

virtually,
Joseph
virtually,
archerja

vonsmith

  • Hero Member
  • *****
  • Posts: 602
    • View Profile
Brain in SQLite?
« Reply #4 on: January 17, 2005, 11:17:51 am »
archerja,
I don't know what to suggest. I haven't investigated SQLite in detail. MySQL might be a better open source choice. We know Robert is pursuing a data base engine for Hal already. Your efforts might not be the best use of your time. However there is much to be learned by experimenting. Personally I plan on seeing what Robert comes up with. In the meantime I hope to study data bases and their application to get a better idea about how to best utilize one for A.I.


=vonsmith=
 

Medeksza

  • Administrator
  • Hero Member
  • *****
  • Posts: 1469
    • View Profile
    • http://www.zabaware.com
Brain in SQLite?
« Reply #5 on: January 18, 2005, 03:11:32 pm »
I've hired someone to work on moving Hal to a database engine. We hit a small snag. Originally I wanted to support Acess, MS SQL, Oracle and mySQL. Here's a quote from the developer:
quote:
In order to count the number of words in one string that are contained in another string, a function is needed within the database (SQL language doesn't support this).  Unfortunately (I never knew this before), you can't call a query that uses an Access VBA function in it from a VB application.  Without that function, it is impossible to count the number of occurrences from SQL.  This only leaves the option of returning them all and counting them the way that you are doing it now, which probably wouldn't be faster than it is now.  This is only a problem with Access, and probably mySQL as well.  SQL Server and Oracle support inline function calls from SQL Statements, so it would work on those systems, but that doesn't solve your problem.


We are now looking into the feasability of using SQLite since it is open source and we can add the required functionality to the engine itself.
Robert Medeksza

vonsmith

  • Hero Member
  • *****
  • Posts: 602
    • View Profile
Brain in SQLite?
« Reply #6 on: January 19, 2005, 10:49:10 am »
Medeksza,
Very interesting point. Thanks for sharing Robert. I wouldn't have predicted this turn of events. However it is still early in this stage of Hal's development. I expect there will be more surprises as you forge ahead.


=vonsmith=
 

recmob

  • Newbie
  • *
  • Posts: 16
    • View Profile
    • http://www.recreationalmobility.com
Brain in SQLite?
« Reply #7 on: January 21, 2005, 06:05:09 pm »
Robert,

I am no programmer in any way, but may I suggest a different opensource database to take a look at? Our website is currently running on PostgreSql (http://www.postgresql.org/). I'd love to see if Hal can be ported to run on this database. I have a development server running if you need someone to test :)))

As it stands now, we have the website and accounting running in PostgreSql, and it would be great to add Hal to this. Jim

Jim Ratte
Recreational Mobility
http://www.recreationalmobility.com


quote:
Originally posted by Medeksza

I've hired someone to work on moving Hal to a database engine. We hit a small snag. Originally I wanted to support Acess, MS SQL, Oracle and mySQL. Here's a quote from the developer:
quote:
In order to count the number of words in one string that are contained in another string, a function is needed within the database (SQL language doesn't support this).  Unfortunately (I never knew this before), you can't call a query that uses an Access VBA function in it from a VB application.  Without that function, it is impossible to count the number of occurrences from SQL.  This only leaves the option of returning them all and counting them the way that you are doing it now, which probably wouldn't be faster than it is now.  This is only a problem with Access, and probably mySQL as well.  SQL Server and Oracle support inline function calls from SQL Statements, so it would work on those systems, but that doesn't solve your problem.


We are now looking into the feasability of using SQLite since it is open source and we can add the required functionality to the engine itself.

Jim Ratte
http://www.recreationalmobility.com
Ultralight / Experimental Aircraft kits & parts

Bill819

  • Hero Member
  • *****
  • Posts: 1483
    • View Profile
Brain in SQLite?
« Reply #8 on: January 23, 2005, 01:23:21 pm »
Hello Joseph
If you have realy found a way to intergrate Hal with that data base why don't you post the coding here on this site so that all of us can benefit from your knowledge? It would be really appreciated by all I am sure. It looks like Mr. M. is going to start using it also so there should not be any problems with have another viewpoint or way of implementing the code.
Thanks from me and the rest of the crowd.
Bill
 

archerja

  • Newbie
  • *
  • Posts: 21
    • View Profile
Brain in SQLite?
« Reply #9 on: January 23, 2005, 10:51:28 pm »
Bill819,

Here is what I did. After installing the wrapper for SQLite(links in above post), I placed a database called "brain.db3" in the main UltraHal folder. The database had a table called "music" that had fields "song", "artist" and "hit_date".  Right now, the database is too big to attach, so I will try to cut it down so that it will fit. I then wrote a very simple script:


'====[beg]======archerja=============
'Hal can use a database
'Look up who sang a song
If InStr(UserSentence, " WHO SANG ") <> 0 Then
   dbq = Trim(Replace(UserSentence, " WHO SANG ", " "))

  BlockSave = True
  GetResponseBlock = True

  If dbq = "" Then
    GetResponse = "I need a song to look for."
  Else
    result = ""
    set oDb = CreateObject( "LiteX.LiteConnection" )
    oDb.Open( "brain.db3" )
    set oStmt = CreateObject( "LiteX.LiteStatement" )
    oStmt.ActiveConnection = oDb
    oStmt.CommandText = "select artist,song from music where song like '%" & dbq & "%'"
    oStmt.Prepare()
    do while not oStmt.Done
   oStmt.Step()
   if oStmt.Done then Exit Do
   result = result & vblf & oStmt.ColumnValue(0) & " sang " & oStmt.ColumnValue(1)
    loop
    If result <> "" then
      GetResponse = result
    Else
      GetResponse = "I do not know that song. Please try another."
    End If
    oStmt.Close()
    oDb.Close()
  End If

End If
'====[end]======archerja=============

Like I said, very simple -- not a lot of error checking.  So, if I ask "Who sang Hotel California?", she will say "Eagles sang hotel california.". I wanted every occurence of a song, so I used "like" instead of "=". I hope this is helpful.

virtually,
Joseph
virtually,
archerja

vonsmith

  • Hero Member
  • *****
  • Posts: 602
    • View Profile
Brain in SQLite?
« Reply #10 on: January 24, 2005, 10:56:23 am »
archerja,
Thanks for the info. Very interesting start. The challenge will be to add new info to the data base through Hal and to search a large number of records for the *best match* response. Finding the best match to the user's inquiry using efficient data base commands is something that we and Zabaware need to figure out.


=vonsmith=
 

Medeksza

  • Administrator
  • Hero Member
  • *****
  • Posts: 1469
    • View Profile
    • http://www.zabaware.com
Brain in SQLite?
« Reply #11 on: January 28, 2005, 01:46:39 am »
Hal's QABrain function has been successfully converted into SQLite. It uses an index and is able to find responses faster. We are now working on converting Hal's TopicSearch, ProcessSubstitutions, ChooseSentenceFromFile, and PatternDB functions into SQLite as well.

Support for KeywordBrain and FragmentMatch will be dropped from SQLite and will no longer be used in future versions of the default brain. Much of the functionality these functions provided will be replaced with a combination of QABrain, TopicSearch, and PatternDB. PatternDB will be used more, it is a powerful function not really utilized in Hal 5.0.

Benefits of moving Hal to SQLite include faster response time, better support in multi-user environments (AIM Bot & Hal Representative), and all data contained in 1 file rather then hundreds of small .brn files. It will also make relationships between different tables easier to control.

The next version of Hal will have reverse compatability with current brains and the current system of text based .brn files. The new default brain however will appear vastly different, but should be easy to pickup for current vbscript brain programmers. Script control will be given for Hal's appointment functions and program launcher. A lot of the complexity of the current script will be moved into the database and dll without sacrificing user control.

The HalBrain editor will be rewritten to support the database. It will allow viewing of the database structure in a "tree" format and will allow jumping from references of a table name in the vbscript directly to the actual table in the database.
Robert Medeksza

onthecuttingedge2005

  • Guest
Brain in SQLite?
« Reply #12 on: January 28, 2005, 02:55:33 am »
quote:
Originally posted by Medeksza

Hal's QABrain function has been successfully converted into SQLite. It uses an index and is able to find responses faster. We are now working on converting Hal's TopicSearch, ProcessSubstitutions, ChooseSentenceFromFile, and PatternDB functions into SQLite as well.

Support for KeywordBrain and FragmentMatch will be dropped from SQLite and will no longer be used in future versions of the default brain. Much of the functionality these functions provided will be replaced with a combination of QABrain, TopicSearch, and PatternDB. PatternDB will be used more, it is a powerful function not really utilized in Hal 5.0.

Benefits of moving Hal to SQLite include faster response time, better support in multi-user environments (AIM Bot & Hal Representative), and all data contained in 1 file rather then hundreds of small .brn files. It will also make relationships between different tables easier to control.

The next version of Hal will have reverse compatability with current brains and the current system of text based .brn files. The new default brain however will appear vastly different, but should be easy to pickup for current vbscript brain programmers. Script control will be given for Hal's appointment functions and program launcher. A lot of the complexity of the current script will be moved into the database and dll without sacrificing user control.

The HalBrain editor will be rewritten to support the database. It will allow viewing of the database structure in a "tree" format and will allow jumping from references of a table name in the vbscript directly to the actual table in the database.



Hi Rob.
Hey, I can't wait to dig into the new version.
Will you be adding support in the new version to allow multiple brains to run simultaniously as a single collective type brain?
Where Brain 1, Brain 2 and Brain 3 can work together for responses
and also have the option to multimix the brains without having to use a single brain if the user wishes to do so? I think this kind of an option would put tons more spice on the whole project.

Load Multibrain Options Button [Load Multiple Brains]

Sincerely.
Gerald L. Blakley

archerja

  • Newbie
  • *
  • Posts: 21
    • View Profile
Brain in SQLite?
« Reply #13 on: January 28, 2005, 11:00:52 am »
Robert Medeksza,

I am very pleased to hear the update!  I can't wait to see it in action.  I know better than to ask for a copy of the database to browse in advance of it going public, but could we maybe get some of the schema (examples of table structures, triggers, views, etc.)?

virtually,
Joseph

virtually,
archerja

spydaz

  • Hero Member
  • *****
  • Posts: 670
    • View Profile
    • http://www.spydazweb.co.uk/
Brain in SQLite?
« Reply #14 on: January 29, 2005, 06:06:24 am »
I do believe FINALLY that we are going in the right direction:

Using SQL give the AI a much more FOCUSED response. ALSO this may improove response times. A BIGGER PROCESSOR / FASTER HARD DRIVE also helps...

In the field of data mining and AI_FUZZY LOGIC.. SQL is also used a an INTERNET search TOOL. Mining the internet for a detailed SEARCHSTRING storing all the relevent data in a database, which is then QUERYED by the user... THIS is closer to "hal serach for information about..." although the response would not be immediate after hal has finished searching :A SET AMOUNT OF TIME " LONG /SHORT. The search data would be added to hals brain. and ready for use in conversation. OR HAL is now ready for QUESTIONING..

This is one of the MAIN features of AI >>>> the ability to have the AI search for information and be ready for questioning RE the data: in a friendly way:

EMOTION is not neccasary at this juncture in time: If concerntration on emotion : THE EMOTIONS WE create/D are not real or even simulated they are just responses...

Emotion can also be a problen as when questioning hal he/she willl go off on a tangent ...

X10
It would be good IF YOU could intergrate X10 into HAL for all those budding HOME AUTOMATION PEOPLE: the protocol is not hard to use as the commands needed to turn an object on is very easy to program, the device to turn on and off could be assigned in the BRAINEDITOR so people could customize control...

I USE AI to TEACH VB and GIVE children a learing tool / talking dictionary / search tool / interactive conversation too.

It would be good IF hal could correct BAD spelling, if he finds the user using BAD SPELLNG hal could correct the user....

WITH SQL / DATAMINING / x10 / HAPTEK CHARACTERS / speech RECOGNITION

we would be able to build much more comprehensive DATABASES of Information into HALS brains... a much more serious task... and not a pornographic CHATBOT as i know some of you are using this feature already..

I DID HAVE A FACE RECOGNITION PROGRAM for hal BUT the Continued poling of the Camera port SLOWS DOWN HAL emensly....

IM WAITING FOR the 64bit processors.....