ps-now

Scrapping Data Using PHP

As part of my PS Now UK Database project I wanted to pull the completion time data from howlongtobeat.com

Scrapping data is something that I had played around with a few years back but I had forgotten how it works.  After an afternoon of research & a lot of trial and error, I managed to create some code that could to that very thing.

The biggest hurdle to get over was pin pointing what the actually search for.  Most websites will give all their tags either a class or id.  In this case, the value I was looking for was inside a plain old <div> tag with no class or ID.  This proved to be a problem.

To understand what data was being pulled from the site I used the print_r() command which showed me I was dealing with a  multi-dimension array.  Something I had not worked with before.  Luckily the print_r() command gives you the locations of each value in the array & using this data I was able to withdraw the data I required.

<pre><?php

function hltb($hltbID) {
    // Function use || to extract the time data from howlongtobeat based upon the game ID stored in the data base
    
    // 1.Get Data based upon hltbID value
    $html = file_get_contents("http://howlongtobeat.com/game.php?id=" . $hltbID);
    
    // howlongtobeat does not name the div that contains the actual time so I will go one div above and then sort through the data
    preg_match_all(
        '/<div>(.*?)<\/div>/s',
        $html, //scrapped file
        $posts,
        PREG_SET_ORDER // formats data into an array of posts
    );
    // 2. Sort  through array to find values needed & print
    echo "<br>Main Story " . $posts[0][1];
    echo "<br>Main + Extras " . $posts[1][1];
    echo "<br>Completionist " . $posts[2][1];
    echo "<br>Combined " . $posts[3][1];
};

include 'conn.php';
// Pull gameID from URL
$newID = $_GET["id"];

    $sql = "SELECT * FROM psnow WHERE gameID=".$newID;
    $result = mysql_query($sql) or die(mysql_error()); 

//Pull the howlongtobeat ID, gameName & gameCount from database
If (mysql_num_rows($result) > 0) {
                while ($row = mysql_fetch_array($result)) {
                    $count = $row['gameCount']+1; // When clicked to view HLTB +1 to count (future use)
                    $gameID = $row['gameID'];
            		echo "<p><b>". $row['gameName'] . "</b>"; //Print Game Name
            		hltb($row['gameHLTB']); //Pass the howlongtobeat game ID into the function
            		
            		//Update the gameCount
            		$update = "UPDATE psnow SET gameCount=" . $count . " WHERE gameID=".$gameID;
            		$result2=mysql_query($update);
		}//end while
}; //end if
?>

Once I was happy with it, I put the code into a function and called it for every single data row in my database (we are talking over 300 entries here).  As you can imagine, the site became slow as it was scrapping data for every entry in my database.  This was not an efficient way of doing things.

So back to the drawing board I went.  To over come this hang time I moved the function into it’s own php file which I then would call into a <div> on the main page by using Jquery.


function loadQueryResults($hltb_ID) {
$('#hltb').load('hltb.php?id=' + $hltb_ID);
$.scrollTo('#right',200);
$('#top10').load('test.php');
return false;
}

It is such a simple script but what it does fixed my problem.  Each entry in my database is a clickable link which has the onClick function added.  When clicked the unique gameID is passed into this function which in turn passes it into the hltb.php file.  Inside that file is the function from before and some extra php to display the game name.  This separate page is then loaded into the the div with the id of htlb.

Problem solved.

desmayer.co.uk/psnow

Numbers

Guess The Number in Python

As I am working through the tutorials helping me learn Python, I am always thinking of how I could use what I have learnt in the real world.  Unless you have a major project the only thing you can do to use what you have learnt is through mini projects.

A quick Google search later & I found an article on Northwestern University Knight Lab that contained five mini projects to carry out using Python.  Reading the list it took me back to some of the example that I use to carry out whilst at University.

Upon this list was a Guess the Number program and I knew that I could complete this project, even at this early stage of learning.

The best way to start any project is to break down what your program is going to do.   So what do I want from my Guess the Number program?

  • Generate random number
  • Prompt for user input
  • Check the value to see if it higher or lower
  • Inform user the number is correct

This is pretty much the basics required to achieve the goal.  I find it best to create a bare bones version of the final program to check that the basic functions work before expanding upon it.  Even with a basic program like this, you can easily add extras on to make it more user friendly;

  • Check input is a Interger
  • Count the number of attempts (including errors)
  • Check how close the number is to the target (warmer & colder)
  • Tell the user if they guess outside the range
  • Inform the user of how many times it took to guess the number

Now with the knowledge of what I want let’s jump into the code.

Declare Variables & Set Up the Random Number

For this mini project I wanted the user to guess a number between 1 & 50.  I hard coded the lower end but I have created the upper end as a variable.  This will allow me to multiple things such as user input for top number & call the variable for output messages.

from random import random, sample
# Start Count
count = 0
# Fix Range
start = int(1)
end = int(51)
# Generate the random number
number = random()
randomNumber = int(number * end)
# Debug - Show number
print('Debug:',randomNumber)

User Input & Check

The next stage is to prompt the user for a number and then check the value entered is a number.

# Enter a Number
print('**** Guess the Number between',start,'and',end-1,' ****')
# Check value is INT
while True:
    try:
        # Repeat the input if statement is false
        guess = int(input('** Enter a Number:'))
    except ValueError:
        # Not an INT
        print('------------------------------')
        print('I said a number!') # Error
        count += 1 # Increase count
        continue
    else:
        # Statement is True break the loop
        break

Using a while True loop to check if the value entered is a number or not. It will keep prompting the user to enter a number until a number has been entered.

Compare Values

These checks will all happen whilst the guess is not equal (!=) to the target value.

As the point of this program is for the user to guess the random number, it will be best to let them know how close they are.  Before that can happen though a check is in place to see if the number entered is higher than the range.  If so, the user is prompted to enter another number.

The next check it to see how close the entered value is to the target.  The check is set up to display “Getting Warmer” when within 5 of the target.  If not, they will get the “Getting Colder” message.

Note: abs(diff) converts the negative value into a positive

Once this check is completed the next will inform the user if they are higher or lower than the target value.  For each time the user guesses wrong (including the integer check), one is added to a count in the background.

At the end of this while loop is a prompt for the user to enter another number as the current value is not equal to the target.  It will keep doing this until the while statement ( guess != randomNumber) gives the result of False.

# While loop until result is right
while guess != randomNumber:
    print('------------------------------')
    if guess > end-1 :
        count += 1
        print('** The range is 1 to',end-1,'- Try Again!')
    else :
        diff = guess - randomNumber
        # print('Debug: ',diff)
        if abs(diff) < 6 :
            print('** Getting Warmer!')
        else:
            print('** Getting Colder!')
        # Check if number is too high
        if guess > randomNumber :
            print('** Too High - Guess Again')
            count += 1
        # Check if number is too low
        elif guess < randomNumber :
            print('** Too Low - Guess Again')
            count += 1
    # Enter value again & Check
    while True:
        try:
            guess = int(input('** Enter a Number:'))
        except ValueError:
            # Not an INT
            print('------------------------------')
            print('I said a number!')
            count += 1
            continue
        else:
            break

Output Result

Once the correct number has been chosen the program will output a message to the user including the number of times it took to guess.

># When the correct number is picked
count += 1
print('------------------------------')
print('** Well done! The number was',randomNumber)
if count == 1 :
    print('** It only took you',count,'attempt!')
else :
    print('** It only took you', count, 'attempts!')

And that is it. A simple program that explores a few basic functions of Python.
Full source code on my github

pillars_of_creation_21848900

Finding Old Work

Boy what a weekend!  It all started late last night when I found my final University project back up on my Google Drive.  A quick transfer to my live server and with a few tweaks I was looking at my final project that I developed 4 years ago!  The only thing that was 100% broken was the Google geocoding but that was due to the updated APIs.  One Sunday morning later and they were all fixed.

What I did find impressive was how well this thing works.  I am not putting myself down here but before starting this project I knew very little about PHP and MySQL.  It was all self taught and when I look back at all the features of this project I must say I am impressed with what I created.

I have spent today looking through the lines of codes working out what I was trying to achieve.  Luckily I did drop a few comments here and there but past me was a very messy coder.  The project had quite a lot going for it but it looks like time ran out during development but as I am topping up my programming skills this year I am going to use this time to expand on what is currently there.

I already have a few ideas for new features for this old project and even started implementing them in the background.

Miuni – www.desmayer.co.uk/miuni

cropped-istock_000011998528medium
new-beginning

Change Starts With You

My name is Des Mayer, a 27 year old family guy and gamer who would be classed as a “bedroom programmer”.  My current work position is so far from my skill set I sometimes wonder how I managed to keep the position.

Since the first time I used a computer I always wanted to understand how they worked and how to manipulate data.  With this in mind during high school I begun to learn how to code.  This took me on a journey through many programming languages including Pascal, VBA.net and JAVA.  Even through the multiple languages I found that I enjoyed solving problems through lines upon lines of code.

Continue reading