dupa

Author Topic: Basics of UltraHal Part II: Editing Database Files  (Read 8342 times)

Don Ferguson

  • Sr. Member
  • ****
  • Posts: 303
    • View Profile
    • http://www.cortrapar.com
Basics of UltraHal Part II: Editing Database Files
« on: September 27, 2003, 04:59:20 pm »
Question:  Can I edit Hal's database files on a text editor?  What do the different patterns in the database files mean?  

Answer:  UltraHal's brain-editing software was developed to reduce the potential of corrupting these files.  However, if you study and are very careful, it is possible to directly edit Hal's database files on a text editor or word processor.  

Turn off auto-correct, spell-check, grammar-check, and other gizmos that might confuse you or cause you to make entries without realizing it.  Have your computer set to show the file-extensions of files, not hide them.  Always make a back-up copy to a different folder before you start, so you can restore the file as it was originally if you make a mistake!

There are two major exceptions to self-editing Hal's files:

1.  The editing of Hal's .dll files is NOT recommended, except for extremely advanced programmers with all the appropriate programming software and training.

2.  User editing of Hal's largest database file, "mainbrain.brn," is no longer possible because it is semi-compiled at Zabaware to speed it up.  This means that the file contains internal cross-references for speed, and it will not work if sections of it get changed.

The information below pertains to editing various ".brn" files.  UltraHal's different "brain" database files have ".brn" file extensions.  The various kinds of files have different data formats.  Here are the most common ones, with advice on each:

RANDOM ENTRY DATABASE FILES

The "random" files simply have words, phrases, or sentences with a line break introducing each one. They often have the word "random" in their names, but some random files have other names, such as "Topic.brn". As the name implies, Hal accesses the file from an external trigger-routine and picks one of the entries at random.

Go into a "random" file with a text editor and the files look like this:

Horses are fine animals.
All mares were once fillies.
Horses need food and exercise.
Meteorites are meteors that struck the ground.
It would be fun to touch a meteorite.

Some entries in random files have a special feature.  They have a flag which holds back part of the entry so that Hal can tell a riddle on one exchange, and the answer on the next.  These can be used for any remark in which we want Hal to "hold back" part two of a message.  These entries look like this:

Why does the chicken cross the road?<TOPIC>Perhaps it's only knowable to a chicken.</TOPIC>
What do you do for a living?<TOPIC>My job is right here with you.</TOPIC>
What is your hobby?<TOPIC>My hobby is chatting.</TOPIC>
I can read your mind.<TOPIC>Right now you are thinking that I can't read your mind.</TOPIC>

It is entirely feasible to write, edit, and add to any and all of the "random" files on any text editor.  It is not likely that you will hurt them.   Do not add blank lines, since they might be selected randomly as responses.

ASSOCIATIVE DATABASE FILES (ALSO CALLED "Q AND A" FILES)

The "associative" files have words, phrases, or sentences in a specific paired-relationship, which is as follows:

@This would be a sentence that Hal might speak about penguins.
THESE WORDS ABOUT PENGUINS MIGHT TRIGGER HAL TO SPEAK THE ABOVE
@Here is another sentence about airplanes that Hal might speak.
WORDS ABOUT AIRPLANES THAT MIGHT TRIGGER HAL TO SPEAK THE ABOVE
@Airplanes are fun to ride on!
GOING PLANE TRIP SOON
@Everybody needs a vacation!
GOING VACATION SOON

(There should be an "@" sign at the beginning of each of the sentence-case sentences above, and each of the ALL-CAPS sentences should begin with one blank space. This forum program removes leading spaces, so we can't show them.)

Notice that in the "associative" files, the potential response sentence is always first, following the "@" sign, and the corresponding user words that might trigger Hal follow.  Also note that the "trigger" lines in caps do not need to be complete sentences.  Compete sentences will work, but so will phrases, sentence fragements, and individual words, depending on what you are trying to associate with the response.

Hal automatically builds these associative databases, relating the user's current statement to Hal's previous statement, and/or to the words in the user's current statement. Some routines add full sentences as triggers, and some routines only use "key words" as triggers.

When Hal uses an associative database, he scans the entire file, evaluates the number of relevant trigger words, the sequence of the trigger words, and selects the match that gets the highest score. If Hal uses the file twice in a row, he randomly selects another close match from the entire file.

Associative files make Hal very powerful because they do NOT require precise matches to work.  As Hal's associative files become more "populated", Hal gets smarter.  Many of Hal's most uncanny and unexpected responses can be generated from these files!

It is possible to write and edit the associative files on any text editor, as long as you maintain the paired-relationship pattern, and don't begin or end the file with any extra line breaks (that would corrupt the file).

Finally, we have...

SPECIFIC RECALL DATABASE FILES (LITERAL STRING-MATCH FILES)

By far the most complicated Hal database format is the specific-recall files. The specific-recall files use literal string matching.  Literal string matching is used by simpler chatterbots, but Hal's are much more complicated.  For instance, the Enhanced_Main.brn file is a specific-recall file.

Hal stores trigger phrases in the specific-recall files, and if an exact character match occurs for one of those phrases, Hal will respond.

Each entry after a line-break in a specific-recall database as actually an "array," which is a set of different items (or "arguments") set apart by commas, which a function can understand and use.

Here is a special annotated example of Hal's specific-recall arrays:

" 1ST TRIGGER PHRASE (REQUIRED) "," OPTIONAL SECOND TRIGGER PHRASE (IF USED, BOTH TRIGGER PHRASES MUST BE FOUND) "," OPTIONAL PHRASE TO CANCEL TRIGGER "," First of three possible Hal responses (required entry). "," Optional second of three possible Hal responses. "," Optional third of three possible Hal responses. ",2,#FALSE#,#FALSE#," Optional additional response words. "

Note that the quotation marks and commas are critical!  If they are missing or out of place, the file will not work!

Here is another example:

" PRETTY "," POODLE "," MIXED BREED "," I like poodles; they're cute; "," Poodle grooming costs money; "," My neighbor has a poodle; ",1,#FALSE#,#FALSE#," "

If the user says, "I have a poodle," this entry returns nothing because both PRETTY and POODLE must be found.

If the user says, "I have a pretty mixed breed poodle," Hal still doesn't trigger because MIXED BREED cancels the call.

If the user says, "I keep my pretty toy poodle in the basement," Hal answers with one of the three phrases.

In the specific-recall databases, the arrays are each separated by a line break. It is possible to write and edit these files on any text editor, but the exact pattern of the arrays MUST be maintained or the file becomes corrupted and unusable!  In the specific-recall brain files, the EXACT usage and number of commas and quotation marks must be maintained!

Also, do not add blank lines to this file.

For those of you who strongly desire to edit the files directly, I hope this is helpful to you!  Back up files and back up often!

P.S. Hal's new versions now have many additional "specialty files" for storing words and phrases. Many of these files have their own unique formats.  If you have a specific question about one of the "specialty files," I will try to answer it.

Sincerely,

Don

« Last Edit: September 27, 2003, 05:29:38 pm by Don Ferguson »
Don Ferguson
E-mail: fergusonrkfd@prodigy.net
Website: www.cortrapar.com
Don's other forum posts: http://www.zabaware.com/forum/search.asp?mode=DoIt&MEMBER_ID=274

Larry

  • Sr. Member
  • ****
  • Posts: 266
    • View Profile
Basics of UltraHal Part II: Editing Database Files
« Reply #1 on: September 27, 2003, 06:35:57 pm »
Don,
Can you help me out with http://www.zabaware.com/forum/topic.asp?TOPIC_ID=818?

I don't see anything in the brain resembling an adverb detector or any simple way to remove them from a temporary sentence. And you seem to be the man to ask hereā€¦

Now let me get back to your post here. I am impressed with your knowledge of Hal's brain. So let me ask you something here. Isn't the topic files the most important files connected with Hal? I'm up to 135 (That's only 13 topic files)... After that most other routines check the length of GetResponse < 4 before they kick in. Of course the user has to use a keyword before they actually kick in...

Thanks for helping us out now and then...
Larry
 

Don Ferguson

  • Sr. Member
  • ****
  • Posts: 303
    • View Profile
    • http://www.cortrapar.com
Basics of UltraHal Part II: Editing Database Files
« Reply #2 on: September 28, 2003, 01:25:33 am »
Hello Larry,

I posted a reply to the other question you referenced, over on the other location.

Regarding your question about the importance of the topic-focus files:

Yes, they are important files, but I would say that all of Hal's routines are intended to work as a team.  If you train Hal consistently and load his databases with lots of statements about lots of subjects, Hal will get smarter in the other databases as well as in the topic-focus databases. It really depends on personal preference and how Hal is trained.  I enjoy hearing Hal pull together ideas across multiple topics, so it doesn't bother me if Hal takes a bit longer to learn and get familiar with a topic.

The single most powerful training activity you can do for Hal is to make dozens and dozens of simple, concise, true STATEMENTS to Hal about every topic that you can think of.  Many users spend most of their time quizzing Hal, or teasing Hal, and Hal learns poorly that way.

Thank you so much for taking the time and interest to learn about Hal and VBS.  All of us Hal fans need as much help as possible to strengthen Hal for the future.

Sincerely,

Don
« Last Edit: September 29, 2003, 01:00:53 am by Don Ferguson »
Don Ferguson
E-mail: fergusonrkfd@prodigy.net
Website: www.cortrapar.com
Don's other forum posts: http://www.zabaware.com/forum/search.asp?mode=DoIt&MEMBER_ID=274

Don Ferguson

  • Sr. Member
  • ****
  • Posts: 303
    • View Profile
    • http://www.cortrapar.com
Basics of UltraHal Part II: Editing Database Files
« Reply #3 on: December 09, 2003, 02:35:27 am »
Hello,

I've received some additional inquiries about the syntax and format of Hal's various brain files.  

The postings above address that issue.  

For those who have sent private e-mails on this subject, this is the thread that I was referring to.

Thank you and best regards,

Don
Don Ferguson
E-mail: fergusonrkfd@prodigy.net
Website: www.cortrapar.com
Don's other forum posts: http://www.zabaware.com/forum/search.asp?mode=DoIt&MEMBER_ID=274

Don Ferguson

  • Sr. Member
  • ****
  • Posts: 303
    • View Profile
    • http://www.cortrapar.com
Basics of UltraHal Part II: Editing Database Files
« Reply #4 on: February 23, 2004, 09:39:17 pm »
Hello,

This is for Padriag and several other persons who have posted or written via e-mail regarding Ultra Hal's various databases.

Hal's databases can be edited a number of ways, including using Wordpad and other word processors.  The main issue is to understand the pattern of the files, and avoid corrupting the file by accidentally changing the pattern.

It is often useful to turn on "show paragraph marks" when using a word processor, so that you KNOW where the line-breaks are!

Also, your computer system should have the "hide file extensions of known file types" turned OFF!  Hiding the file extensions causes mass confusion on Windows machines.  You can find this option by going (on Windows XP) to Windows Explorer, tools, folder options, view, and scroll down to "Hide file extensions for known file types." UNCHECK this box if it has a check mark in it.

Sincerely,

Don
« Last Edit: February 23, 2004, 09:44:04 pm by Don Ferguson »
Don Ferguson
E-mail: fergusonrkfd@prodigy.net
Website: www.cortrapar.com
Don's other forum posts: http://www.zabaware.com/forum/search.asp?mode=DoIt&MEMBER_ID=274

vonsmith

  • Hero Member
  • *****
  • Posts: 602
    • View Profile
Basics of UltraHal Part II: Editing Database Files
« Reply #5 on: February 23, 2004, 11:31:04 pm »
Great mini-tutorial Don,
I've been looking at the specific-recall files lately. Some of the phrase detection and sentence parsing I plan on doing will be getting tedious for what I have in mind. I've experimented with a number of different methods in the XTF brain. I think it's time for me to try out the specific-recall method.

I'm still a little unclear about a few details though. What does the sequence ",2,#FALSE#,#FALSE#," mean compared to ",1,#FALSE#,#FALSE#,"? What do entries with #TRUE# in one or both positions mean?

Thanks for the info,


=vonsmith=
« Last Edit: February 23, 2004, 11:32:03 pm by vonsmith »
 

recmob

  • Newbie
  • *
  • Posts: 16
    • View Profile
    • http://www.recreationalmobility.com
Basics of UltraHal Part II: Editing Database Files
« Reply #6 on: February 24, 2004, 09:55:45 pm »
quote:
Originally posted by Don Ferguson

Hello,

This is for Padriag and several other persons who have posted or written via e-mail regarding Ultra Hal's various databases.



Don,

Have you worked with the Representative version? I'm curious if I could teach Hal on a local computer what I would like him to know about our products, then drop the brain files into the Representative version at a later date (on my own server). If not, I'll have to ante up and purchase it long before I'll put it to actual use :)

I asked Vonsmith a few days ago, he replied he hadn't worked with the Rep version. I also emailed Robert directly, but have not yet received a response. Jim
« Last Edit: February 24, 2004, 10:01:24 pm by recmob »
Jim Ratte
http://www.recreationalmobility.com
Ultralight / Experimental Aircraft kits & parts

Don Ferguson

  • Sr. Member
  • ****
  • Posts: 303
    • View Profile
    • http://www.cortrapar.com
Basics of UltraHal Part II: Editing Database Files
« Reply #7 on: February 26, 2004, 01:52:16 am »
Hello Recmob,

Two quick definitions for new forum members who are reading this:

There is a FREE Ultra Hal Representative which runs on Zabaware's server, and can be set up to seem like a page on our various individual web sites.  The free version has a 500K database-size limit.

There is a PAID version of Ultra Hal representative which is software that runs on the owner's own server.  The paid version is under the owner's control and has an unlimited database size.

I don't own the paid version of Ultra Hal Representative.  The Ultra Hal on my site is the free version, running on the Zabaware server.  For that reason, the only file editing that I do is through the Zabaware web interface.  I have never seen the native database file to know how it is formatted.

It sounds feasible in theory that you could amass a large database in advance on a local computer, and then "transfer" it to the paid Ultra Hal representative at a later date after you buy it.  What you would need to know (and I DON'T know) is, "What is the database file format used in the paid version of Ultra Hal Representative?"

Once you obtained a sample of that file format for Zabaware, it seems reasonable that you could use a word processor and populate the database file, by hand, in advance of purchasing the software.

Sincerely,

Don
Don Ferguson
E-mail: fergusonrkfd@prodigy.net
Website: www.cortrapar.com
Don's other forum posts: http://www.zabaware.com/forum/search.asp?mode=DoIt&MEMBER_ID=274

Don Ferguson

  • Sr. Member
  • ****
  • Posts: 303
    • View Profile
    • http://www.cortrapar.com
Basics of UltraHal Part II: Editing Database Files
« Reply #8 on: February 26, 2004, 02:13:08 am »
Hello Vonsmith,

Question:

I'm still a little unclear about a few details though. What does the sequence ",2,#FALSE#,#FALSE#," mean compared to ",1,#FALSE#,#FALSE#,"?

Answer:

The numbers are priority numbers within the database file.  "1" has the highest priority.  A "2" will only trigger if a "1" was not found.  A "3" will only trigger if neither a "1" nor a "2" was found.  A "4" will only trigger if neither a "1" nor a "2" nor a "3" was found.

Question:

What do entries with #TRUE# in one or both positions mean?

Answer:

TRUE in the first position is a flag to turn on a special user-sentence-fragment repeat function.  It causes Hal to respond with his answer, followed by the portion of the user sentence following the keyword, followed by the string of characters at the very end of the database entry.  An example follows:

Suppose the user is named Ralph and he says, "I am going to climb a mountain."

Of course, this pronoun-reverses to "You are going to climb a mountain."

Suppose Hal had the following database entry:

" YOU ARE GOING TO ","",""," You sound determined.  Let me know when you  ","","",1,#TRUE#,#FALSE#," , okay, <UserName>? "

The user's sentence matches the trigger string, so Hal will use this database entry to respond.  The "TRUE" flag tells Hal to use the special function.

Hal's response would be, "You sound determined.  Let me know when you climb a mountain, okay, Ralph?"

Look in the Enhanced_Main.brn database, and you can see quite a few entries with this flag set to TRUE.  Also, it's used with the RUNIT /RUNIT tags, to "trap" an application name and put it between the two tags to trigger the opening of an application or file.

Regarding the second-position TRUE-FALSE flag:  I have never seen a database entry with TRUE in the second position.  I don't know what it does.  I suspect that Zabaware built it into the database structure for some future purpose, which we haven't seen yet.

Sincerely,

Don
Don Ferguson
E-mail: fergusonrkfd@prodigy.net
Website: www.cortrapar.com
Don's other forum posts: http://www.zabaware.com/forum/search.asp?mode=DoIt&MEMBER_ID=274

recmob

  • Newbie
  • *
  • Posts: 16
    • View Profile
    • http://www.recreationalmobility.com
Basics of UltraHal Part II: Editing Database Files
« Reply #9 on: February 26, 2004, 08:14:21 am »
quote:
Originally posted by Don Ferguson

Hello Recmob,

Two quick definitions for new forum members who are reading this:

There is a FREE Ultra Hal Representative which runs on Zabaware's server, and can be set up to seem like a page on our various individual web sites.  The free version has a 500K database-size limit.

There is a PAID version of Ultra Hal representative which is software that runs on the owner's own server.  The paid version is under the owner's control and has an unlimited database size.

I don't own the paid version of Ultra Hal Representative.  The Ultra Hal on my site is the free version, running on the Zabaware server.  For that reason, the only file editing that I do is through the Zabaware web interface.  I have never seen the native database file to know how it is formatted.

It sounds feasible in theory that you could amass a large database in advance on a local computer, and then "transfer" it to the paid Ultra Hal representative at a later date after you buy it.  What you would need to know (and I DON'T know) is, "What is the database file format used in the paid version of Ultra Hal Representative?"

Once you obtained a sample of that file format for Zabaware, it seems reasonable that you could use a word processor and populate the database file, by hand, in advance of purchasing the software.

Sincerely,

Don



Don,

Thanks very much for your reply. One reason I'm asking this is working with the web interface I've had some trouble using the free version. I don't know if Hal learns by just entering small statements.

In the web interface (under change account options), the first question is "Is your representative allowed to learn from people that talk to it? (People would be able to override your current brain entries, and teach it false information)" When I change this to yes (or any options on that page), and click store options they are not actually saved at all. If I don't make Hal live to others while I teach Hal, I don't see this as a problem (Hal learning from conversations with me). Manually entering every question and possible answer through the Teach Hal button would be a nightmare :) Jim
Jim Ratte
http://www.recreationalmobility.com
Ultralight / Experimental Aircraft kits & parts

Don Ferguson

  • Sr. Member
  • ****
  • Posts: 303
    • View Profile
    • http://www.cortrapar.com
Basics of UltraHal Part II: Editing Database Files
« Reply #10 on: February 26, 2004, 04:34:04 pm »
Hi Jim,

I haven't tried changing my Representative settings via the web interface lately, but it sounds like yours is malfunctioning for you.  It should save your settings.  I can only speculate that, because Zabaware's server traffic has been rising rapidly, something might not be working the way it is supposed to be.

As far as allowing "real time" learning versus "manual" learning on the web Representative Hal:

At first I permitted real-time learning.  It surprised and delighted some users.  However, it had a downside.  Some users would spout fifty lines of vulgarity and gibberish, and Hal Rep would of course immediately "learn" their vulgarity and gibberish.  For that reason, I had to do maintenance to edit the database at least twice a day, to prevent Hal from saying bad things to innocent users.

Since then, I've shut off the "real-time" learning.  The "manual" teaching using the web interface isn't as labor-intensive as you might think, once you get the pattern of it!  Whenever you spot a user remark in a transcript for which you'd like to teach Hal, just click "Change Hal's Answer" and the input screen appears, showing the user remark and a blank space for the new Hal answer.  Type in your answer, click the button, and presto!

Now here's something additional that might not be obvious.  Suppose you want to give Hal two or three different answers to the same user input for a bit of variety.  After you click the teach button, and you get the confirmation that you've taught Hal, just click your web browser's "back" button, and you're back at the same teaching screen with the same user remark, and a new blank form.  Enter another Hal answer, click teach again, and now you've taught Hal two different answers!  You can do this as many times as you wish!

Here are the main things that I like about using the web interface and the "manual" teach mode:

--Hal won't learn gibberish and vulgarity.

--Hal is fairly easy to teach, and using the "back" button, you can give Hal variety in his responses.

--By teaching Hal responses to actual user input, you conserve the efficiency with the database, while making sure that your responses are to actual remarks that real users make.  That prevents you from working for hours loading remarks for trigger-phrases that actual site visitors never say.

Keep examining the Zabaware Hal Representative web interface, and I think you'll find that you can gain a lot more speed and proficiency than you might think at first.  It's really quite a handy interface.  Definitely let Zabaware know whenever the interface doesn't act as if it's functioning properly.

Sincerely,

Don
Don Ferguson
E-mail: fergusonrkfd@prodigy.net
Website: www.cortrapar.com
Don's other forum posts: http://www.zabaware.com/forum/search.asp?mode=DoIt&MEMBER_ID=274

recmob

  • Newbie
  • *
  • Posts: 16
    • View Profile
    • http://www.recreationalmobility.com
Basics of UltraHal Part II: Editing Database Files
« Reply #11 on: February 26, 2004, 05:37:48 pm »
quote:
Originally posted by Don Ferguson

Hi Jim,

I haven't tried changing my Representative settings via the web interface lately, but it sounds like yours is malfunctioning for you.  It should save your settings.  I can only speculate that, because Zabaware's server traffic has been rising rapidly, something might not be working the way it is supposed to be.

As far as allowing "real time" learning versus "manual" learning on the web Representative Hal:

At first I permitted real-time learning.  It surprised and delighted some users.  However, it had a downside.  Some users would spout fifty lines of vulgarity and gibberish, and Hal Rep would of course immediately "learn" their vulgarity and gibberish.  For that reason, I had to do maintenance to edit the database at least twice a day, to prevent Hal from saying bad things to innocent users.

Since then, I've shut off the "real-time" learning.  The "manual" teaching using the web interface isn't as labor-intensive as you might think, once you get the pattern of it!  Whenever you spot a user remark in a transcript for which you'd like to teach Hal, just click "Change Hal's Answer" and the input screen appears, showing the user remark and a blank space for the new Hal answer.  Type in your answer, click the button, and presto!

Now here's something additional that might not be obvious.  Suppose you want to give Hal two or three different answers to the same user input for a bit of variety.  After you click the teach button, and you get the confirmation that you've taught Hal, just click your web browser's "back" button, and you're back at the same teaching screen with the same user remark, and a new blank form.  Enter another Hal answer, click teach again, and now you've taught Hal two different answers!  You can do this as many times as you wish!

Here are the main things that I like about using the web interface and the "manual" teach mode:

--Hal won't learn gibberish and vulgarity.

--Hal is fairly easy to teach, and using the "back" button, you can give Hal variety in his responses.

--By teaching Hal responses to actual user input, you conserve the efficiency with the database, while making sure that your responses are to actual remarks that real users make.  That prevents you from working for hours loading remarks for trigger-phrases that actual site visitors never say.

Keep examining the Zabaware Hal Representative web interface, and I think you'll find that you can gain a lot more speed and proficiency than you might think at first.  It's really quite a handy interface.  Definitely let Zabaware know whenever the interface doesn't act as if it's functioning properly.

Sincerely,

Don



Don,

I certainly agree with you about the real time learning from others and teaching it vulgar items. I had planned to switch it to real time learning only while I teach it what I want it to know. The link would be a hidden link on the website, not viewable to others. I'd then turn off the real time learning and put the 'help' link viewable on the website to others.

I feel that if I could at least teach Hal the basics and let him learn from my statements, I can then tweak his responses as you suggested. I have played with that already, although I hadn't thought of the back button trick. Thanks!

Currently we carry almost 20,000 items in our store, and many relate to each other. For instance one aircraft kit uses Rotax brand aircraft engines, which relate to propeller recommendations for that kit, engine maintenance, etc.

I big project I know, but since no one else in this industry has implented this kind of help in their websites, perhaps I could gain more of the market from the bigger companies and reduce the time my phone is firmly planted to my ear :))

I did send a note to Robert M. this morning about the web interface not saving changes. I also tried starting a new account, and changes made to that one were not saved either. Perhaps he has the save function disabled. I'll post what response I get from him when I receive it. Jim
Jim Ratte
http://www.recreationalmobility.com
Ultralight / Experimental Aircraft kits & parts

vonsmith

  • Hero Member
  • *****
  • Posts: 602
    • View Profile
Basics of UltraHal Part II: Editing Database Files
« Reply #12 on: February 27, 2004, 12:13:34 am »
Don,
Thanks very much for the info on the specific-recall files and usage. It's a big help.


=vonsmith=