What Is a Neural Network?

It's a technique for building a computer program that learns from data. It is based very loosely on how we think the human brain works. First, a collection of software "neurons" are created and connected together, allowing them to send messages to each other. Next, the network is asked to solve a problem, which it attempts to do over and over, each time strengthening the connections that lead to success and diminishing those that lead to failure. For a more detailed introduction to neural networks, Michael Nielsen's Neural Networks and Deep Learning is a good place to start. For a more technical overview, try Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.

This Is Cool, Can I Repurpose It?

Please do! We've open sourced it on GitHub with the hope that it can make neural networks a little more accessible and easier to learn. You're free to use it in any way that follows our Apache License. And if you have any suggestions for additions or changes, please let us know.

We've also provided some controls below to enable you tailor the playground to a specific topic or lesson. Just choose which features you'd like to be visible below then save this link, or refresh the page.

What Do All the Colors Mean?

Orange and blue are used throughout the visualization in slightly different ways, but in general orange shows negative values while blue shows positive values.

The data points (represented by small circles) are initially colored orange or blue, which correspond to positive one and negative one.

In the hidden layers, the lines are colored by the weights of the connections between neurons. Blue shows a positive weight, which means the network is using that output of the neuron as given. An orange line shows that the network is assiging a negative weight.

In the output layer, the dots are colored orange or blue depending on their original values. The background color shows what the network is predicting for a particular area. The intensity of the color shows how confident that prediction is.

What Library Are You Using?

We wrote a tiny neural network library that meets the demands of this educational visualization. For real-world applications, consider theTensorFlow library.

 

[Memo]Tensorflow: how to save/restore a model?

In(and after) Tensorflow version 0.11:

Save the model:

import tensorflow as tf

#Prepare to feed input, i.e. feed_dict and placeholders

w1 = tf.placeholder("float", name="w1")

w2 = tf.placeholder("float", name="w2")

b1= tf.Variable(2.0,name="bias")

feed_dict ={w1:4,w2:8}

#Define a test operation that we will restore

w3 = tf.add(w1,w2)

w4 = tf.multiply(w3,b1,name="op_to_restore")

sess = tf.Session()

sess.run(tf.global_variables_initializer())

#Create a saver object which will save all the variables

saver = tf.train.Saver()

#Run the operation by feeding input

print sess.run(w4,feed_dict)

#Prints 24 which is sum of (w1+w2)*b1 

#Now, save the graph

saver.save(sess, 'my_test_model',global_step=1000)

Restore the model:

import tensorflow as tf

sess=tf.Session()    

#First let's load meta graph and restore weights

saver = tf.train.import_meta_graph('my_test_model-1000.meta')

saver.restore(sess,tf.train.latest_checkpoint('./'))

# Access saved Variables directly

print(sess.run('bias:0'))

# This will print 2, which is the value of bias that we saved

# Now, let's access and create placeholders variables and

# create feed-dict to feed new data

graph = tf.get_default_graph()

w1 = graph.get_tensor_by_name("w1:0")

w2 = graph.get_tensor_by_name("w2:0")

feed_dict ={w1:13.0,w2:17.0}

#Now, access the op that you want to run. 

op_to_restore = graph.get_tensor_by_name("op_to_restore:0")

print sess.run(op_to_restore,feed_dict)

#This will print 60 which is calculated 

This and some more advanced use-cases have been explained very well here.

http://cv-tricks.com/tensorflow-tutorial/save-restore-tensorflow-models-quick-complete-tutorial/

 

 

For TensorFlow version < 0.11.0RC1:

The checkpoints that are saved contain values for the Variables in your model, not the model/graph itself, which means that the graph should be the same when you restore the checkpoint.

Here's an example for a linear regression where there's a training loop that saves variable checkpoints and an evaluation section that will restore variables saved in a prior run and compute predictions. Of course, you can also restore variables and continue training if you'd like.

x = tf.placeholder(tf.float32)

y = tf.placeholder(tf.float32)

w = tf.Variable(tf.zeros([1, 1], dtype=tf.float32))

b = tf.Variable(tf.ones([1, 1], dtype=tf.float32))

y_hat = tf.add(b, tf.matmul(x, w))

...more setup for optimization and what not...

saver = tf.train.Saver()  # defaults to saving all variables - in this case w and b

with tf.Session() as sess:

    sess.run(tf.initialize_all_variables())

    if FLAGS.train:

        for i in xrange(FLAGS.training_steps):

            ...training loop...

            if (i + 1) % FLAGS.checkpoint_steps == 0:

                saver.save(sess, FLAGS.checkpoint_dir + 'model.ckpt',

                           global_step=i+1)

    else:

        # Here's where you're restoring the variables w and b.

        # Note that the graph is exactly as it was when the variables were

        # saved in a prior training run.

        ckpt = tf.train.get_checkpoint_state(FLAGS.checkpoint_dir)

        if ckpt and ckpt.model_checkpoint_path:

            saver.restore(sess, ckpt.model_checkpoint_path)

        else:

            ...no checkpoint found...

        # Now you can run the model to get predictions

        batch_x = ...load some data...

        predictions = sess.run(y_hat, feed_dict={x: batch_x})

Here are the docs for Variables, which cover saving and restoring. And here are the docs for the Saver.

 

 

You can also take this easier way.

Step.1 - Initialize all your variables

W1 = tf.Variable(tf.truncated_normal([6, 6, 1, K], stddev=0.1), name="W1")

B1 = tf.Variable(tf.constant(0.1, tf.float32, [K]), name="B1")

 

Similarly, W2, B2, W3, .....

Step.2 - Save the list inside Model Saver and Save it

model_saver = tf.train.Saver()

 

# Train the model and save it in the end

model_saver.save(session, "saved_models/CNN_New.ckpt")

Step. 3 - Restore the model

with tf.Session(graph=graph_cnn) as session:

    model_saver.restore(session, "saved_models/CNN_New.ckpt")

    print("Model restored.")

    print('Initialized')

Step. 4 - Check Variable

W1 = session.run(W1)

print(W1)

---

While running in different python instance, use

with tf.Session() as sess:

    # Restore latest checkpoint

    saver.restore(sess, tf.train.latest_checkpoint('saved_model/.'))

 

    # Initalize the variables

    sess.run(tf.global_variables_initializer())

 

    # Get default graph (supply your custom graph if you have one)

    graph = tf.get_default_graph()

 

    # It will give tensor object

    W1 = graph.get_tensor_by_name('W1:0')

 

    # To get the value (numpy array)

    W1_value = session.run(W1)

 

 

In most cases, saving and restoring from disk using a tf.train.Saver is your best option:

... # build your model

saver = tf.train.Saver()

with tf.Session() as sess:

    ... # train the model

    saver.save(sess, "/tmp/my_great_model")

with tf.Session() as sess:

    saver.restore(sess, "/tmp/my_great_model")

    ... # use the model

You can also save/restore the graph structure itself (see the MetaGraph documentation for details). By default, the Saver saves the graph structure into a .meta file. You can call import_meta_graph() to restore it. It restores the graph structure and returns a Saver that you can use to restore the model's state:

saver = tf.train.import_meta_graph("/tmp/my_great_model.meta")

with tf.Session() as sess:

    saver.restore(sess, "/tmp/my_great_model")

    ... # use the model

However, there are cases where you need something much faster. For example, if you implement early stopping, you want to save checkpoints every time the model improves during training (as measured on the validation set), then if there is no progress for some time, you want to roll back to the best model. If you save the model to disk every time it improves, it will tremendously slow down training. The trick is to save the variable states to memory, then just restore them later:

... # build your model

# get a handle on the graph nodes we need to save/restore the model

graph = tf.get_default_graph()

gvars = graph.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)

assign_ops = [graph.get_operation_by_name(v.op.name + "/Assign") for v in gvars]

init_values = [assign_op.inputs[1] for assign_op in assign_ops]

with tf.Session() as sess:

    ... # train the model

    # when needed, save the model state to memory

    gvars_state = sess.run(gvars)

    # when needed, restore the model state

    feed_dict = {init_value: val

                 for init_value, val in zip(init_values, gvars_state)}

    sess.run(assign_ops, feed_dict=feed_dict)

A quick explanation: when you create a variable X, TensorFlow automatically creates an assignment operation X/Assign to set the variable's initial value. Instead of creating placeholders and extra assignment ops (which would just make the graph messy), we just use these existing assignment ops. The first input of each assignment op is a reference to the variable it is supposed to initialize, and the second input (assign_op.inputs[1]) is the initial value. So in order to set any value we want (instead of the initial value), we need to use a feed_dict and replace the initial value. Yes, TensorFlow lets you feed a value for any op, not just for placeholders, so this works fine.