Mdthbs

Multiband vertical antenna not working as expected

Was planting UN flag on Moon ever discussed?

Why did the World Bank set the global poverty line at $1.90?

Could a person damage a jet airliner - from the outside - with their bare hands?

Rail-to-rail op-amp only reaches 90% of VCC, works sometimes, not everytime

C++ logging library

How can powerful telekinesis avoid violating Newton's 3rd Law?

Extracting data from Plot

Do you have to have figures when playing D&D?

Why is Na5 not played in this line of the French Defense, Advance Variation?

What differences exist between adamantine and adamantite in all editions of D&D?

Was Self-modifying-code possible just using BASIC?

Why is the length of the Kelvin unit of temperature equal to that of the Celsius unit?

Make Gimbap cutter

How (un)safe is it to ride barefoot?

How do we say "within a kilometer radius spherically"?

Proving that a Russian cryptographic standard is too structured

Is it safe to remove python 2.7.15rc1 from Ubuntu 18.04?

Assigning function to function pointer, const argument correctness?

Increase speed altering column on large table to NON NULL

Razzle Dazzle simulator

Cathode rays and the cathode rays tube

Can the removal of a duty-free sales trolley result in a measurable reduction in emissions?

Does the Nuka-Cola bottler actually generate nuka cola?

Is the Keras Embedding layer dependent on the target label?

How does Keras 'Embedding' layer work?Should word embedding vectors be normalized before being used as inputs?deep learning - word embedding with parts of speechRandomly initialized embedding matrixHow to use Keras pre-trained 'Embedding' layer?How the embedding layer is trained in Keras Embedding layerDimension reduction - word embeddings as inputs for a time series model (LSTM)What is difference between keras embedding layer and word2vec?Learning image embeddings using VGG and Word2VecCan an embedding layer be replaced by a fully connected layer?Autoencoder keeping constant vector as predict in keras

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

1

$begingroup$

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the Keras Embedding layer maps distinct categorical features to n-dimensional vectors, which allows us to find, for example, how similar two features are.

What I do not understand is how these vectors in the embedding layer are trained. Here is an explanation where there is information that these vectors are not computed with any operation, but working only as a lookup table, but I always thought that they are somehow "trained" to find similarities between distinct features.

If they are trained, are they trained from target labels, or from the order in which they appear (similar to GloVe, word2vec, etc.) or from both?

I have the following example of two pairs of rows in a dataset. y is the model target label and X are the features encoded to integers to be used in the embedding layer:

#pair 1    

dataset_y_row1 = [1]

dataset_y_row2 = [0]

dataset_X_row1 = [3,5,8,45,2]

dataset_X_row2 = [3,5,8,45,2]



#pair 2

dataset_y_row3 = [1]

dataset_y_row4 = [1]

dataset_X_row3 = [3,5,8,45,2]

dataset_X_row4 = [3,5,45,8,2]

My questions are the following:

1. Will the embedding layer see any difference between rows 1 and 2 (i.e. is
   it 'target-label-sensitive')?


2. Will the embedding layer see any difference between rows 3 and 4 (i.e. is it sensitive to order of features like word2vec, GloVe, etc.)?

neural-networks keras word-embeddings embeddings

share|cite|improve this question

edited 7 hours ago

Mihai Chelaru

1877

asked 11 hours ago

Jan MusilJan Musil

354

$endgroup$

add a comment | 

1

$begingroup$

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the Keras Embedding layer maps distinct categorical features to n-dimensional vectors, which allows us to find, for example, how similar two features are.

What I do not understand is how these vectors in the embedding layer are trained. Here is an explanation where there is information that these vectors are not computed with any operation, but working only as a lookup table, but I always thought that they are somehow "trained" to find similarities between distinct features.

If they are trained, are they trained from target labels, or from the order in which they appear (similar to GloVe, word2vec, etc.) or from both?

I have the following example of two pairs of rows in a dataset. y is the model target label and X are the features encoded to integers to be used in the embedding layer:

#pair 1    

dataset_y_row1 = [1]

dataset_y_row2 = [0]

dataset_X_row1 = [3,5,8,45,2]

dataset_X_row2 = [3,5,8,45,2]



#pair 2

dataset_y_row3 = [1]

dataset_y_row4 = [1]

dataset_X_row3 = [3,5,8,45,2]

dataset_X_row4 = [3,5,45,8,2]

My questions are the following:

1. Will the embedding layer see any difference between rows 1 and 2 (i.e. is
   it 'target-label-sensitive')?


2. Will the embedding layer see any difference between rows 3 and 4 (i.e. is it sensitive to order of features like word2vec, GloVe, etc.)?

neural-networks keras word-embeddings embeddings

share|cite|improve this question

edited 7 hours ago

Mihai Chelaru

1877

asked 11 hours ago

Jan MusilJan Musil

354

$endgroup$

add a comment | 

$begingroup$

I learned how to 'use' the Keras Embedding layer, but I am not able to find any more specific information about the actual behavior and training process of this layer. For now, I understand that the Keras Embedding layer maps distinct categorical features to n-dimensional vectors, which allows us to find, for example, how similar two features are.

What I do not understand is how these vectors in the embedding layer are trained. Here is an explanation where there is information that these vectors are not computed with any operation, but working only as a lookup table, but I always thought that they are somehow "trained" to find similarities between distinct features.

If they are trained, are they trained from target labels, or from the order in which they appear (similar to GloVe, word2vec, etc.) or from both?

I have the following example of two pairs of rows in a dataset. y is the model target label and X are the features encoded to integers to be used in the embedding layer:

#pair 1    

dataset_y_row1 = [1]

dataset_y_row2 = [0]

dataset_X_row1 = [3,5,8,45,2]

dataset_X_row2 = [3,5,8,45,2]



#pair 2

dataset_y_row3 = [1]

dataset_y_row4 = [1]

dataset_X_row3 = [3,5,8,45,2]

dataset_X_row4 = [3,5,45,8,2]

My questions are the following:

1. Will the embedding layer see any difference between rows 1 and 2 (i.e. is
   it 'target-label-sensitive')?


2. Will the embedding layer see any difference between rows 3 and 4 (i.e. is it sensitive to order of features like word2vec, GloVe, etc.)?

neural-networks keras word-embeddings embeddings

share|cite|improve this question

edited 7 hours ago

Mihai Chelaru

1877

asked 11 hours ago

Jan MusilJan Musil

354

$endgroup$

#pair 1    

dataset_y_row1 = [1]

dataset_y_row2 = [0]

dataset_X_row1 = [3,5,8,45,2]

dataset_X_row2 = [3,5,8,45,2]



#pair 2

dataset_y_row3 = [1]

dataset_y_row4 = [1]

dataset_X_row3 = [3,5,8,45,2]

dataset_X_row4 = [3,5,45,8,2]

share|cite|improve this question

edited 7 hours ago

Mihai Chelaru

1877

asked 11 hours ago

Jan MusilJan Musil

354

share|cite|improve this question

edited 7 hours ago

Mihai Chelaru

1877

asked 11 hours ago

Jan MusilJan Musil

354

edited 7 hours ago

Mihai Chelaru

1877

edited 7 hours ago

Mihai Chelaru

1877

asked 11 hours ago

Jan MusilJan Musil

354

asked 11 hours ago

Jan MusilJan Musil

354

1 Answer
1

active

oldest

votes

3

$begingroup$

Embeddings layer for vocabulary of size $m$, that encodes each word into embeddings vector of size $k$ is a shorthand for having the words one-hot encoded using into $m$ features and then putting dense layer with $k$ units over it. Word2vec and GloVe are specialized algorithms for learning the embeddings, but the end product is a matrix of weights that is multiplied by the one-hot encoded words.

If you are interested in detailed, yet accessible introductory source on word embeddingss, check the series of blog post by Sebastian Ruder .

To answer your question, one would need to consider what is your network architecture and the data. Algorithms like word2vec and GloVe are trained on language data, to predict things like next word in a sequence. On another hand, if you use the embeddingss layer that is trained from the scratch and used as a part of larger network, that has some utilitarian purpose (e.g. spam detection, sentiment classification), then the layers work as any other dense layers, so they serve purpose of automatic feature engineering. In the latter case, you would expect to see more specialised embeddingss, that would learn features related to the objective of your network.

share|cite|improve this answer

edited 8 hours ago

answered 10 hours ago

Tim♦Tim

61.9k9136234

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  okay, thanks just ask to "but the end product is a matrix of weights that is multiplied by the one-hot encoded words." This is related to word2vec and glove, or also to the first part of paragraph (keras Embedding layer). Does it mean that Embedding vector of size m can be just simulated by using one hot encoded layer as input, and dense layer with m neurons? So vector for each one-hot encoded feature should be just it's m weights going from this input feature to dense layer neurons?
  $endgroup$
  – Jan Musil
  9 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @JanMusil as I said, embeddingss are dense layers, so they are matrices of weights to be multiplied by the features, it applies to all the embeddings.
  $endgroup$
  – Tim♦
  8 hours ago
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f412206%2fis-the-keras-embedding-layer-dependent-on-the-target-label%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

3

$begingroup$

Embeddings layer for vocabulary of size $m$, that encodes each word into embeddings vector of size $k$ is a shorthand for having the words one-hot encoded using into $m$ features and then putting dense layer with $k$ units over it. Word2vec and GloVe are specialized algorithms for learning the embeddings, but the end product is a matrix of weights that is multiplied by the one-hot encoded words.

If you are interested in detailed, yet accessible introductory source on word embeddingss, check the series of blog post by Sebastian Ruder .

To answer your question, one would need to consider what is your network architecture and the data. Algorithms like word2vec and GloVe are trained on language data, to predict things like next word in a sequence. On another hand, if you use the embeddingss layer that is trained from the scratch and used as a part of larger network, that has some utilitarian purpose (e.g. spam detection, sentiment classification), then the layers work as any other dense layers, so they serve purpose of automatic feature engineering. In the latter case, you would expect to see more specialised embeddingss, that would learn features related to the objective of your network.

share|cite|improve this answer

edited 8 hours ago

answered 10 hours ago

Tim♦Tim

61.9k9136234

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  okay, thanks just ask to "but the end product is a matrix of weights that is multiplied by the one-hot encoded words." This is related to word2vec and glove, or also to the first part of paragraph (keras Embedding layer). Does it mean that Embedding vector of size m can be just simulated by using one hot encoded layer as input, and dense layer with m neurons? So vector for each one-hot encoded feature should be just it's m weights going from this input feature to dense layer neurons?
  $endgroup$
  – Jan Musil
  9 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @JanMusil as I said, embeddingss are dense layers, so they are matrices of weights to be multiplied by the features, it applies to all the embeddings.
  $endgroup$
  – Tim♦
  8 hours ago
  

  
  
  
  

add a comment | 

3

$begingroup$

Embeddings layer for vocabulary of size $m$, that encodes each word into embeddings vector of size $k$ is a shorthand for having the words one-hot encoded using into $m$ features and then putting dense layer with $k$ units over it. Word2vec and GloVe are specialized algorithms for learning the embeddings, but the end product is a matrix of weights that is multiplied by the one-hot encoded words.

If you are interested in detailed, yet accessible introductory source on word embeddingss, check the series of blog post by Sebastian Ruder .

To answer your question, one would need to consider what is your network architecture and the data. Algorithms like word2vec and GloVe are trained on language data, to predict things like next word in a sequence. On another hand, if you use the embeddingss layer that is trained from the scratch and used as a part of larger network, that has some utilitarian purpose (e.g. spam detection, sentiment classification), then the layers work as any other dense layers, so they serve purpose of automatic feature engineering. In the latter case, you would expect to see more specialised embeddingss, that would learn features related to the objective of your network.

share|cite|improve this answer

edited 8 hours ago

answered 10 hours ago

Tim♦Tim

61.9k9136234

$endgroup$

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  $begingroup$
  okay, thanks just ask to "but the end product is a matrix of weights that is multiplied by the one-hot encoded words." This is related to word2vec and glove, or also to the first part of paragraph (keras Embedding layer). Does it mean that Embedding vector of size m can be just simulated by using one hot encoded layer as input, and dense layer with m neurons? So vector for each one-hot encoded feature should be just it's m weights going from this input feature to dense layer neurons?
  $endgroup$
  – Jan Musil
  9 hours ago
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  @JanMusil as I said, embeddingss are dense layers, so they are matrices of weights to be multiplied by the features, it applies to all the embeddings.
  $endgroup$
  – Tim♦
  8 hours ago
  

  
  
  
  

add a comment | 

$begingroup$

Embeddings layer for vocabulary of size $m$, that encodes each word into embeddings vector of size $k$ is a shorthand for having the words one-hot encoded using into $m$ features and then putting dense layer with $k$ units over it. Word2vec and GloVe are specialized algorithms for learning the embeddings, but the end product is a matrix of weights that is multiplied by the one-hot encoded words.

If you are interested in detailed, yet accessible introductory source on word embeddingss, check the series of blog post by Sebastian Ruder .

To answer your question, one would need to consider what is your network architecture and the data. Algorithms like word2vec and GloVe are trained on language data, to predict things like next word in a sequence. On another hand, if you use the embeddingss layer that is trained from the scratch and used as a part of larger network, that has some utilitarian purpose (e.g. spam detection, sentiment classification), then the layers work as any other dense layers, so they serve purpose of automatic feature engineering. In the latter case, you would expect to see more specialised embeddingss, that would learn features related to the objective of your network.

share|cite|improve this answer

edited 8 hours ago

answered 10 hours ago

Tim♦Tim

61.9k9136234

$endgroup$

share|cite|improve this answer

edited 8 hours ago

answered 10 hours ago

Tim♦Tim

61.9k9136234

answered 10 hours ago

Tim♦Tim

61.9k9136234

answered 10 hours ago

Tim♦Tim

61.9k9136234