Hal's new XTF Brain is finally released.

[HALImprover]

I ran into the 'too many file' problem because I am using the auto-idle feature which allows me to send sentences to Hal automatically every 5 seconds or so. Every instance of the script is creating new files and this causes Hal to hit that file limit very quickly.

 I am almost positive that this error originates from the fact that the Operating System sets a limit on the number of files that can be set or retained in memory at one time. If a file is opened by a program it is placed in memory and referenced by the File System as well as the program which accessed it (meaning Hal). When Hal is done with the file it is still referenced by the file system (and considered open even though the file is no longer accessed) until it is closed either by Hal or the Operating System. I believe that the main Hal program does not release the file from memory when finished. I've run tests in VB6 with the same code I use in Hal and I don't get the 'too many files' error. I recreated the appendtofile function in VB6, adding a line to close the file when finished.

 The same can be said about the HalBrain object that is created during the brain script execution. When the script is finished the object still exists in memory, except the reference to that object is lost once the script loses scope. I recommend putting a line at the end of each function in the script which sets the object to nothing.
eg. Set HalBrain = Nothing
This frees the object from memory after each use, ultimately preventing Hal from gradually uses more and more memory. If you want some proof on that note, try using the auto-chat code I previously posted and leave it run for more than a few hours. What will eventually happen is Windows will try to increase the virtual memory (to meet the increasing demands of Hal) and wind up crashing the Hal program in the process. Setting the object to nothing will reduce the likelihood of this problem occuring, but will not entirely prevent it. I remember reading several topics on this forum regarding errors arising from long conversation sessions with Hal.

 I would suggest using Notepad to edit the .brn files because I find MS Word much slower and it blocks writes to files which MS Word has open. Notepad can access the files without blocking Hal from reading/writing to them.

 I did some investigating and discovered the answer to my first question. When the TopicSearch function is used Hal searches the entire user sentence for each word or phrase from the defined .brn file starting from the bottom of the file.

 Thanks for your insight into this problem Don, it is much appreciated. I would like to get into some related issues and problems, but I'm afraid this may end up becoming somewhat of a novella if I continue any further. More to come in future posts!

 Happy coding, and remember. Don't take life too seriously. [:p] You never make it out alive! []

 PS. Thank you as well vonsmith for the advice. I'm sure if it wasn't for the kind suggestions and courteous replies on this forum I would have given up on Hal a long time ago. I plan on repaying these gestures in the near future with some of my labors on Hal.
 By the way, I have a question about your XTF brain's topic search ability. I understand in general how the process works and I have to say that we have struck a similar note in our search for the topic in the user's sentence. I too use a static list of linking verbs to refine the topic search. I search for the first occurance of the first linking verb and check if a topic is present within a sentence split based on that linking verb. If no valid topic exists than I continue with the next linking verb, and so on. The actual topic search is defined in a function which searches for 6 word topics, 5 word topics, etc until a topic is found (if possible). My question is does your script test for every linking verb and logically choose the topic after or does it find the first linking verb which produces results and use that topic?

[vonsmith]

HALImprover,
The XTF Brain approach doesn't depend just on linking verbs to split a sentence. The first part of the XTF function searches for over 800 types sentence fragments. I theorized that the phrases that were found "bounded" within a sentence fragment would not just contain a topic, but had a high likelyhood of being a very specific multi-word topic. A few of the fragments are: "The * are" and "That * could", "Those * might, etc.

If a fragment is found then the * represents a topic phrase. Poor topic words like "a", "at", "do", "she" are filtered out. Adverbs like "frequently" or "mostly" are filtered out. Whatever is left over is most often either a single word or multi-word topic.

If a double-ended fragment isn't found then the XTF function looks for single-ended fragments. The prior examples were double-ended. Single-ended examples are: "The *" or "* are". I found that single-ended fragments aren't reliable enough to derive multi-word topics, but work well identifying a single word topic.

Poor topic words include all pronouns. I tried including pronouns as topics, but found pronouns are too general. Hal thought nearly every sentence topic was "I this" or "You that" or whatever pronoun. Sentence subjects don't necessarily make good topics. Example: "You like pretty horses." "You" wouldn't make a good topic, use "horse" instead. I also found that the best topic is sometimes found in the object phrase of the sentence.

The XTF Brain gets pretty complicated after this point. Basically the XTF function takes processed sentence fragments and searches for known multi-word topics first since a multi-word topic like "big red horse" is much more "topic focused" than just "horse". When a new topic is found it is saved to one of several XTF topic files. If no multi-word topics are found then single-word topics are searched for by the method I discussed in my previous post. For both multi and single-word topics an attempt is made to match to the PrevTopic (previous sentence topic) in the XTF function. Hal tries to stay on the same topic on each exchange with the user. The XTF Brain also uses the Wordnet function to find "related" words to use as flags to decide whether or not to stay with the previous topic. The XTF function can also "learn" new related words for future reference.

I have posted some more detail about the XTF function elsewhere on the forum and have included a summary in the Instruction Manual packaged with the XTF Brain download. The best method to understand the XTF function is to use the <dbxtfon> command in the XTF Brain. That command directs all of the debug info collected in the XTF function (just within the function, not the entire brain) to be saved in a file Z_XTF_DEBUG_LOG.TXT. That debug info can be used to "walk through" the operation of the XTF function.

I hope this provides a little insight. However this just barely scrapes the surface of the XTF function.


=vonsmith=

[drmweaver2]

vonsmith asked<<Is there any case where any of those prepositions or prepositional phrases would immediately precede the pronoun "I" in any typical person's sentence? I haven't identified any such cases yet.>>

How about these?
After I went, she came by.
As I sew, so shall I weep.
But I want some.
Considering I want it now....
Down I fell. ((Down functions as an adverb here --or as part of the verbal phrase "to fall down")
Like I said, she stank at pool.
Nearby, I went to school. ((The comma would surely tip this off as "nearby" not functioning as a preposition.))
Off I went on her.. ((another verbal phrase... to go off on))
On I went. ((another verbal phrase... to go on))
Out I went. ((another verbal phrase... to go out))
Outside I went. ((another verbal phrase... to go outside))
Over I fell. ((another verbal phrase... to fall over))
Round and round I went.
Save I, all were killed.
Since I know better than he does....
He knows better than I.
Throughout, I watched with wonder. ((The comma would surely tip this off as "nearby" not functioning as a preposition.))
Till I went home, I didn't know what I had left behind.
Up I went. ((another verbal phrase... to go up))
Within I went, then again, without. (an obsolete English form. Now probably only seen in poetic usage.)

[vonsmith]

drmweaver2,
Thanks for the excellent examples. These are the kind of examples I need to work out a detection methodology. I don't know what can be done about people's poor use of grammar, poetic style or lack of punctuation. The outdated forms are rarely used so that shouldn't be an issue. The remaining forms will need some work to find a solution. Thanks.


=vonsmith=

[Don Ferguson]

Hello Dreamweaver,

Thank you for excellent examples of cases in which Hal would improperly substitute "me" for "I" due to the particular structure of certain sentences, such as:

After I drove...
Before I began...
Underneath I hurried...
On I continued...

These are definitely constructions that I didn't consider in the original build-up of the content that went into the "corrections.brn" and "substitutions.brn."

For MOST (but not all) the cases shown, it will be possible to correct the problem with additional entries that "reverse the incorrect reversals" and thus take care of the less-usual constructions.  

I described how this would be done in an earlier posting on this thread.

The additional entries in the "substitutions.brn" and "corrections.brn" would be LOWER IN THE LISTS than the existing substitution-pairs, so that they would be processed later, and thus have the "last word."

The entries would look like the following:

"me begin","I begin"
"me began","I began"
"me continue","I continue"
"me continued,"I continued"
"me drive","I drive"
"me drove","I drove"
"me hurry","I hurry"
"me hurried,"I hurried"

Please note that the convention of form that I started in the original databases was to list both the present and past tenses of verbs, and to put most of the list in alphabetical order.  That made it easier during subsequent manual editing to avoid the waste of duplicate entries.

A few hundred entries would cover 99% of cases, and add only a very tiny amount of processing work for Hal.

I sense that many of us tend to want to invent a rule to "tell" Hal and let him figure these things out for himself; however, in this case I haven't yet found a substitute for just sitting down for a couple of hours and putting in the labor of making the list.  

I will be happy to do so, after I get past a couple of current projects.

Sincerely,

Don

[drmweaver2]

[)]No problem providing the "examples". Took about 5 minutes to write them all.

Speaking of which, the "together me" issue is an interesting one. I have spent the day seriously trying to come up with some example sentences, any sentences with this combintation in that order. I can't unless there is a comma between the two words. I also tried looking on the Net for that pairing of words, including doing a LExus-Nexus search of News sources - no joy there either except when the two are are separated by a comma or other punctuation mark.

In most cases the punctuation mark is being used as a delimiter between clauses or to set off "a list" from the rest of the sentence.

Personally, I never thought of "together" as a single-word preposition but am much more familiar with it as part of two-word prepositional phrases like "together with".

I'll keep looking around and thinking


Side note...After I get some more experience with HAL itself, I might actually try coding something (I only downloaded it last week and have spent the weekend loading him up with "common sense quotations" and knowledge of one subject area. Until I gt some experience with the program results, I probably can't suggest help with any real coding except in this type of "language discussion".)

[drmweaver2]

A few posts earlier, vonsmith wrote:
<<Sentence subjects don't necessarily make good topics. Example: "You like pretty horses." "You" wouldn't make a good topic, use "horse" instead. I also found that the best topic is sometimes found in the object phrase of the sentence.>>

What is happening here, at least linguistically-speaking, is that you've "re-discovered" the passive voice in a sense.
I like horses. - active
Horses are liked by me - passive

However, by switching to the "object" of the active voice sentence as the "topic", you have actually changed the "focus" of the sentence entirely.

In my admitted ignorance of the code structure, I have to ask whether or not this is what is actually intended considering that HAL incorporates a "remembered response" learning style. By that I mean, we are not speaking about horses so much in the active example above as we are talking about "me". On the other hand, the focus IS on the horses in the passive version.

So, what would be updated using the passivized subject-topic file is different than what would be updated using the active-subject interpretation. This actually might lead to a "loss of information" as far as training HAL goes.

Consider the case where I am trying to describe myself to HAL more than trying to describe properties of horses to HAL. See? IF HAL uses the passive interpretation, does he update the "me" file also or does the information get "lost" in the horses topic file?

Which brings me to the statement written earlier in the same paragraph, <<Poor topic words include all pronouns. I tried including pronouns as topics, but found pronouns are too general.>>...

Pronouns have a linguistic quality of "reference" that nouns tend not to have in the same manner. As pointed out, without the "reference" nearby, pronouns seem to dangle abstractly - even for humans trying to interpret them. The general exceptions are I/me and you/we. These are normally pretty clear in terms of reference.

The problem of abstraction may be limited to 3rd person pronouns - he/she/it/they if coded stringently.

I don't know if that "helps" or is just rambling out of ignorance, but I thought I would throw it out for consideration. [)]

If I am speaking out of turn here, I apologize in advance.

[vonsmith]

drmweaver2,
You've made some interesting points that touch on the difference between humans and Hal. Hal isn't able to discriminate on fine points of sentence structure. Most of the time Hal uses a loose method of word association to pick a sentence from his topic files to use as a response to the user's sentence. That association capability gives the impression that Hal is "thinking" about that topic and directly responding to the user.

Your examples:
I like horses. - active
Horses are liked by me - passive

In both cases it is desireable for Hal to use "horse" as the topic. Typical conversations with Hal often includes pronouns. Any pronoun, if used as a topic, would obstruct Hal's ability to stay on topic. If the topic was "I" (from Hal's perspective) then everything would be too narrowly focused on Hal.

Examples:
I like cookies.
I should go to the store.
I think bricks are used to build walls.
Brain surgery is what I need.

If Hal focused on the pronoun "I" as a topic then all of the above sentences would be stored in one big "I" topic file. Hal would tend to response only with "I" focused responses instead of more general knowledge.

Excluding the "I" pronoun above causes Hal to file the topics, "cookies", "store", "bricks" or "wall" and "surgery" into their respective topic files. The information is saved to the proper topic file most of the time. Some topic files necessarily have a small percentage of unrelated info in them. Hal is just a computer after all. However for Hal it doesn't matter much whether the topic resides in the subject or object of a sentence. What chatbots need to do is be able to chain related thoughts together with the user.

Of course we want Hal to "know" something of himself and of the user. For example, special cases "you are", "i am", "your" or similar constuctions can be handled by chatbots separately from the general topic focus. Hal has some capability along those lines already. I'm programming some new functions that will give Hal an improved ability to discriminate things about himself or the user. Handling individual special cases separately allows for easier programming.


=vonsmith=

[drmweaver2]

I think that, to a small extent, I am suggesting that at least certain "I am"-type statements (where an active voice focus suggests that the "real world focus" in a statement is on the speaker and not on HAL, might warrant a special function/some special coding.

That coding would/could "double enter" - make entries in the passivized subject-topic files as well as in speaker-specific files. I have noted that as speakers/users change, there are specific knowledge items that are not universal. I think I confirmed this by identifying files for another user than myself after I let someone else play with HAL and I looked at the DEFbrain directory. But I could be wrong.

The question is not really whether it is possible - of that I am sure. The question is whether it is appropriate to consider for future modifications as HAL will supposedly be used by more than one user in most installations --- people will most likely want to show off their "creations" in the most inoccuous example. Should larger plans for HAL be in the user's plan, I would think that multiple users with respect to a particular HAL brain is inevitable and thus, updating specific user-knowledge files would be desirable.

Not a big deal. Mostly a theoretic one at this point I guess.

[drmweaver2]

After rereading my post, I left out the real point I wanted to make.

The examples you (vonsmith) gave are exactly what the type of thing I was thinking of.
I/We aren't interested in cookies per se, in the first example - at least in my case. Instead, I would like HAL to know *I* like this or that type of cookie and that I get them from the store (perhaps).[:p] If you draw HAL's response from a generic "cookie" topic file, then that information is lost/not present in HAL (if it was ever typed in by the user).[]

I definitely agree that <<What chatbots need to do is be able to chain related thoughts together with the user>>. Perhaps I just don't know enough about the "engine" yet.[?]

[vonsmith]

drmweaver2,
The original Hal brain saves generic (non-pronoun containing) information to "shared" topic files. That information is used with all users that log on to Hal. The XTF Brain goes one step further. Generic information is stored in shared user files, however Hal specific information is also stored there too. This way a little piece of Hal's personality is shared among users. Multiple users each contribute a little bit to Hal's personality. From Hal's perspective, copies of sentences containing "I", "me", "my and "mine" end up in the shared files.

I agree with your statement about "I like" special cases. As I am typing this I am sitting at my home computer writing script to address "I like" type sentences. I'm trying to write it in a general way so that the concept can be extended to other special case phrases while reusing the same core code. The method I'm using should preserve the relationship of Hal to his "like".


=vonsmith=

[Morlhach]

strange [!]

Hal with the XTF 1.2 brain crashes with the sentence "Give a person a fish and you feed them for a day; teach a person to use the Internet and they won't bother you for weeks."

[?]

[NT Canuck]

quote:Originally posted by Morlhach

strange [!]

Hal with the XTF 1.2 brain crashes with the sentence "Give a person a fish and you feed them for a day; teach a person to use the Internet and they won't bother you for weeks."

[?]

I tried with and without the ; (subbed a ,), both ran ok here.
P4 2.6ghz (HTT), 1gb ram.

NTCanuck: Give a person a fish and you feed them for a day, teach
a person to use the Internet and they won't bother you for weeks.
Crystal: Try to pay an honest compliment to every person, every day.

NTCanuck: Give a person a fish and you feed them for a day; teach
a person to use the Internet and they won't bother you for weeks
Crystal: Any day spent with me is a worthwhile day.

'Seek and ye shall find'
NT Canuck
http://ntcanuck.com

[vonsmith]

Morlhach,
I tried many variations of the sentences provided. My Hal XTF Brain v1.2 installation worked fine. Does the crash occur each and every time when that sentence is used? Have you pasted any other scripts into the XTF Brain v1.2 or made other modifications?

Also make certain you have renamed the "enhanced_main.brn" as shown in this posting:
http://www.zabaware.com/forum/topic.asp?TOPIC_ID=1237

Not renaming the file won't cause a crash, but some Hal functions won't work without it.

Let us know how your Hal is doing.


=vonsmith=

[Bill819]

Greetings
A question for you. Is your new brain a stand-a-lone? Does it have to be intergrated into the existing Hal brain? In either case is it compatible with Don's brain modifications? Can I load both and combine them? Do I ask to many questions? ha ha
Bill
[]