Intelligent Webscraper MRK I (Prototype)

[Spitfire2600]

Hello everyone again,

After the catastrophic debut of my ConceptNetMRKIII plugin (which I've got an update for to clean up yet) I have written a plugin based on ConceptNet to scrape Wikipedia for acceptable related answers. Simply ask a question and Hal will run 1 of 37 protocols based on keywords, 1st nouns, secondary nouns, related nouns, related wiki topics, and alternate phrasing. It's still a little hit and miss, about 60-70% accurate as of now with questions like "What is Nevada's population" or "Does Atlantis exist"


Installation:

Drag and drop all contents of zip to Hal's Directory, which should be in "Program files (x86)/Zabaware/Ultra Hal 7/" Other wise you'll have to change it in the UHP on line 20.
You may need to create a sentence table in Hals brain named "webscrape" if Hal doesn't, but he really should.



And done! Feel free to improve, I have a couple of things I am also going to add into this just to enhance the search. A file called WebScrape8.txt will appear in your WebScraper Folder. This is for debugging which terms were searched. Each number corresponds with a term protocol in the code from lines 1734 and 2551 ("m working on simplifying this series of processes)



Additional weird notes I'm apparently not smart enough to figure out:

I thought I had every combination of term, but clearly I'm missing something because this does miss some very obvious questions sometimes. Feel free to take a look.

Sometimes looking at "Webscrape8.txt" Hal will research terms twice together, "Chicken Chicken 17." I'm not sure why. This shouldn't be allowed but Hal's going to do it anyway. I may have added too many protocols at once, call me ambitious.



I believe with some work, Robert and Hal could really make use of this and ConceptNet together (Not mine, MIT's)

This is intended for CPU's with a higher Ghz +2.8 to really not take forever to find information.

-Spitfire2600


**Note** Updated to Full Mrk I Version***

[lightspeed]

Very interesting , this sound like something i mentioned before about using many (from list ) of "various " sites (inspiration, life, poems, etc. etc. and based on subject do a random info pull of a web site to the best one , so hal would answer as a more intelligent a.i.  , i had mentioned this to doonycam (who is now no longer here) and to cyberjedi,  as i mentioned i think doing what i mentioned by many various site links and a random pick of the various such websites by best subject pick , is the key to making hal respond a lot smarter .   
for what my opinion is worth  "if"  something like this is done i personally would like to see twp settings , or script where a person can have a specific ask and get answer or a second one that anytime their is a subject that hal will access a website and talk about it (after answering what the  user has originally talked about) .   
I will download this newest plug in and try it out . i'll un check all others to make sure nothing interferes with it.

[lightspeed]

Okay i tried it out and ask a question what is a pedicar  , should i have put quotation marks around the subject name?  anyway what happened is i got a error 424 on line 3066 in column 1. object required "obj fso"
i went into the debugscript file and it showed this :






i placed the folder and uhp of yours in the hal 7 zabaware folder in programs (x86).

[lightspeed]

p.s. it also gave a blank answer , and when i just typed hello later gave another blank answer.

 later on   hal error just as mentioned above same one popped up again . 

[Spitfire2600]

Ah yes, I see. I forgot to set FSO.

I have corrected this error and it should be good to use now. Thanks!

-Spitfire2600

[Spitfire2600]

There is no need for punctuation. The webscraper will remove these. Just make sure that the Webscraper folder from the zip file is in your Hal Directory as well and it should be good to go with this new update.

-Spitfire2600

[lightspeed]

Okay , you may have posted this else where but can you give a rundown of all this is suppose to do ,  (if you have else where either link it or just copy and paste the info back on here please. )  also any do or don'ts of how to ask , some you just mentioned.  also did you see what i mentioned about many line list of various web  sites  etc., on hal accessing many sites and randomly choosing by subject etc. ?

[lightspeed]

OKAY have ran into another problem , when i asked a question now i got a new error . here is the new hal error ( also hal said he could not locate that information. ) it did it again on another question i asked , how tall is the empire state building .

does a question have to have a question mark? for this plug in to work . i didn't use one the second time but i think i did on the first question i had asked , also something popped up and said something like an application was busy and do i want to switch  or something like that .
below is  the new error .

[Spitfire2600]

Error -214Something is an Internet Explorer issue.  This is likely a conflict with another plugin.

[lightspeed]

only other plug in i have checked is gender and age .

[lightspeed]

i am not sure why but the questions i have asked aren't being answered , i have asked how old does a red tree grow to be . i also asked , hoiw tall is the empire state building .   i also asked how many inches are in a yard . and each time this plug in only said : I could not located that information. 

[Spitfire2600]

lightspeed,
My original post basically sums up what this plug-in does.

This plug-in will take a question such as "What is the population of Arizona?"

Break it into its constituents like a keyword and first or second nouns, so "Population" is the first noun, "Arizona" is the second noun.
 
It will then start a chain protocol to determine the correct answer is found. All answers come from Wikipedia. So first it will search "Population" for the term "Arizona" if no answer is found, it will continue, breaking the sentence apart further the more complex the question is. For this question, the protocol should stop at searching "Arizona" for any reference to "Population." It is generally fairly effective.

Toward the end of the protocol, the code will invert keywords, grab synonyms of all terms, and grab related wiki article terms if no answer is found.

Sometimes Hal will hang, giving a message like "Application is busy" or something like that.... Just let it go, Hal is still searching, it's just that time timeout code is very short. Hal is perfectly fine, just busy. On fast computers, Hal shouldn't take longer than 20-30 seconds to find something for any general question.

If Hal can't find an answer, simply rephrase or change terms. I am working on this in future releases if everyone can confirm this prototype does work.

-Spitfire2600

[Art]

Have you (or anyone) tried it with other Plug-ins selected and if so, were there any issues like with Doonycram or other ones?

(The reason I mentioned it was that  Doonycram's S.A.L.L.I.E. plugin was quite large and was comprised of a host of other/different Plugins). Might = conflicts?

Thanks!

[Spitfire2600]

Plug-ins I have tried for compatibility with no issues. VRFreewill, Markov, ConceptNet, SmartAlarm.

-Spitfire2600

[lightspeed]

WHEN I TRIED IT I ONLY HAD the age and gender plug in checked . and tried it even without that one . if the question isn't being answered  as is i wonder if additional ways can be written in .  as i asked in a straight forward question , as if for instance anyone would do with siri , or Alexa , i think it's more a matter of re writing ways or more ways of questions in form of being asked .