Zabaware Support Forums

Zabaware Forums => Ultra Hal Assistant File Sharing Area => Topic started by: Spitfire2600 on March 25, 2018, 01:21:29 am

Title: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 25, 2018, 01:21:29 am
Hello everyone again,

After the catastrophic debut of my ConceptNetMRKIII plugin (which I've got an update for to clean up yet) I have written a plugin based on ConceptNet to scrape Wikipedia for acceptable related answers. Simply ask a question and Hal will run 1 of 37 protocols based on keywords, 1st nouns, secondary nouns, related nouns, related wiki topics, and alternate phrasing. It's still a little hit and miss, about 60-70% accurate as of now with questions like "What is Nevada's population" or "Does Atlantis exist"


Installation:

Drag and drop all contents of zip to Hal's Directory, which should be in "Program files (x86)/Zabaware/Ultra Hal 7/" Other wise you'll have to change it in the UHP on line 20.
You may need to create a sentence table in Hals brain named "webscrape" if Hal doesn't, but he really should.



And done! Feel free to improve, I have a couple of things I am also going to add into this just to enhance the search. A file called WebScrape8.txt will appear in your WebScraper Folder. This is for debugging which terms were searched. Each number corresponds with a term protocol in the code from lines 1734 and 2551 ("m working on simplifying this series of processes)



Additional weird notes I'm apparently not smart enough to figure out:

I thought I had every combination of term, but clearly I'm missing something because this does miss some very obvious questions sometimes. Feel free to take a look.

Sometimes looking at "Webscrape8.txt" Hal will research terms twice together, "Chicken Chicken 17." I'm not sure why. This shouldn't be allowed but Hal's going to do it anyway. I may have added too many protocols at once, call me ambitious.



I believe with some work, Robert and Hal could really make use of this and ConceptNet together (Not mine, MIT's)

This is intended for CPU's with a higher Ghz +2.8 to really not take forever to find information.

-Spitfire2600


**Note** Updated to Full Mrk I Version***
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 26, 2018, 10:13:40 am
Very interesting , this sound like something i mentioned before about using many (from list ) of "various " sites (inspiration, life, poems, etc. etc. and based on subject do a random info pull of a web site to the best one , so hal would answer as a more intelligent a.i.  , i had mentioned this to doonycam (who is now no longer here) and to cyberjedi,  as i mentioned i think doing what i mentioned by many various site links and a random pick of the various such websites by best subject pick , is the key to making hal respond a lot smarter .   
for what my opinion is worth  "if"  something like this is done i personally would like to see twp settings , or script where a person can have a specific ask and get answer or a second one that anytime their is a subject that hal will access a website and talk about it (after answering what the  user has originally talked about) .  :) 
I will download this newest plug in and try it out . i'll un check all others to make sure nothing interferes with it.
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 26, 2018, 11:03:07 am
Okay i tried it out and ask a question what is a pedicar  , should i have put quotation marks around the subject name?  anyway what happened is i got a error 424 on line 3066 in column 1. object required "obj fso"
i went into the debugscript file and it showed this :






i placed the folder and uhp of yours in the hal 7 zabaware folder in programs (x86).
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 26, 2018, 11:14:17 am
p.s. it also gave a blank answer , and when i just typed hello later gave another blank answer.

 later on   hal error just as mentioned above same one popped up again .  :(
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 26, 2018, 01:58:31 pm
Ah yes, I see. I forgot to set FSO.

I have corrected this error and it should be good to use now. Thanks!

-Spitfire2600
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 26, 2018, 02:01:17 pm
There is no need for punctuation. The webscraper will remove these. Just make sure that the Webscraper folder from the zip file is in your Hal Directory as well and it should be good to go with this new update.

-Spitfire2600
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 26, 2018, 05:49:05 pm
Okay , you may have posted this else where but can you give a rundown of all this is suppose to do ,  (if you have else where either link it or just copy and paste the info back on here please. )  also any do or don'ts of how to ask , some you just mentioned.  also did you see what i mentioned about many line list of various web  sites  etc., on hal accessing many sites and randomly choosing by subject etc. ?
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 26, 2018, 07:17:27 pm
OKAY have ran into another problem , when i asked a question now i got a new error . here is the new hal error ( also hal said he could not locate that information. ) it did it again on another question i asked , how tall is the empire state building .

does a question have to have a question mark? for this plug in to work . i didn't use one the second time but i think i did on the first question i had asked , also something popped up and said something like an application was busy and do i want to switch  or something like that .
below is  the new error .
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 26, 2018, 08:41:02 pm
Error -214Something is an Internet Explorer issue.  This is likely a conflict with another plugin.
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 26, 2018, 08:47:45 pm
only other plug in i have checked is gender and age .
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 26, 2018, 09:17:32 pm
i am not sure why but the questions i have asked aren't being answered , i have asked how old does a red tree grow to be . i also asked , hoiw tall is the empire state building .   i also asked how many inches are in a yard . and each time this plug in only said : I could not located that information.  :(
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 26, 2018, 09:26:59 pm
lightspeed,
My original post basically sums up what this plug-in does.

This plug-in will take a question such as "What is the population of Arizona?"

Break it into its constituents like a keyword and first or second nouns, so "Population" is the first noun, "Arizona" is the second noun.
 
It will then start a chain protocol to determine the correct answer is found. All answers come from Wikipedia. So first it will search "Population" for the term "Arizona" if no answer is found, it will continue, breaking the sentence apart further the more complex the question is. For this question, the protocol should stop at searching "Arizona" for any reference to "Population." It is generally fairly effective.

Toward the end of the protocol, the code will invert keywords, grab synonyms of all terms, and grab related wiki article terms if no answer is found.

Sometimes Hal will hang, giving a message like "Application is busy" or something like that.... Just let it go, Hal is still searching, it's just that time timeout code is very short. Hal is perfectly fine, just busy. On fast computers, Hal shouldn't take longer than 20-30 seconds to find something for any general question.

If Hal can't find an answer, simply rephrase or change terms. I am working on this in future releases if everyone can confirm this prototype does work.

-Spitfire2600
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Art on March 27, 2018, 08:07:42 am
Have you (or anyone) tried it with other Plug-ins selected and if so, were there any issues like with Doonycram or other ones?

(The reason I mentioned it was that  Doonycram's S.A.L.L.I.E. plugin was quite large and was comprised of a host of other/different Plugins). Might = conflicts?

Thanks!
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 27, 2018, 10:59:19 am
Plug-ins I have tried for compatibility with no issues. VRFreewill, Markov, ConceptNet, SmartAlarm.

-Spitfire2600
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 27, 2018, 02:18:12 pm
WHEN I TRIED IT I ONLY HAD the age and gender plug in checked . and tried it even without that one . if the question isn't being answered  as is i wonder if additional ways can be written in .  as i asked in a straight forward question , as if for instance anyone would do with siri , or Alexa , i think it's more a matter of re writing ways or more ways of questions in form of being asked .
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 27, 2018, 09:56:19 pm
Is the webscraper folder from the zip files located in your Hal Directory? Is your Hal Directory location defined properly in the UHP? Did Hal create the Sentence table mentioned in my first post?

I asked "how old does a red tree grow to be" which was one of your questions and I did get a response back from Wikipedia "For the wealthy, the dye used was kermes, made from a tiny scale insect which fed on the branches and leaves of the oak tree."

-Spitfire2600
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: lightspeed on March 29, 2018, 12:42:08 pm
that answer is not correct "how old  about a red tree " instead it pulled up apparently about a dye being used . "
Title: Re: Intelligent Webscraper MRK I (Prototype)
Post by: Spitfire2600 on March 29, 2018, 08:41:09 pm
This is why it's beta. It is finding several results, however, I need to revamp the filters so it chooses the correct reference.

-Spitfire2600